I typically solve this problem in one of two ways.  I categorize machines, host or guest, as either needing Java or not needing Java and then maintain the installation.  Second, on the machines that need it, I will use more than one browser and only enable Java in one of the browser flavors.  Typically on a Windows system I will allow Java to connect to IE, and use a plugin like Quick Java to disable Java in FireFox. With Java 7u10, they introduced a check box to disable the browser functionality in the Java Control Panel.  I don’t like this approach as there are places that need Java (I have kids who like Minecraft).  I prefer to have it so only one browser can access Java, and that is the secondary browser only used for that purpose.

I don’t see uninstalling Java to be a reasonable solution.  It is not the language itself that is the problem, it is the connection to the browser that allows people with evil intent to be able to run arbitrary code when a vulnerability is discovered.  Just like binary malware, a user would have to actively execute a JAR file to get Java to run code when not invoked by the browser.  We don’t see much of that because it requires user interaction.  What we often see is a Java exploit as part of a drive-by attack, where the only purpose of the Java code is to grab a piece of malware written and compiled for Windows and invoke it on the system.

If you don’t need Java, then by all means, get rid of it.  It is overhead on the system and it only exposes the system to exploitation if it is forgotten about and not patched (Yes I know 0-days mean even patched and maintained are exposed).  But a lot of people do have legitimate uses for Java.  Most of these uses don’t involve the browser and they cannot remove Java unless they are willing to give up that use.  For them, they need to either disconnect Java from all browsers, or develop and stick to a safe usage plan.

I do wish that Java would auto-update and did not require admin privileges to do so.  As stated, my children like Minecraft, but they don’t have admin accounts.  That means they cannot update Java on a Windows machine even if it were to notify them there is a Java update. I need to find a way like in Ubuntu where I can add the update command to the list of commands that do not require root to invoke. Alternatively, Oracle could make the Java install able to be set to automatically download and update security patches by the installing user (that would be admin/root at time of install) and run as a service at that user level.  My only fear there is I might get some unwanted Ask.com toolbars after an update.

Also, to give some kudos to Oracle, Java 7u11 does notify the user of Java content trying to run as applets in a browser and ask them to confirm they want it to run.  This is great if systems are fully patched.  My experience is that most are not because there is no auto-update feature, images with old installs are used to push out production machines, or the user only updates it when they use Java.

The concept is to build a stand alone Java application that will allow the incident responder to keep multiple observations open.  The observations will be editable, but have full change tracking after initial save.  I chose Java because it is the easiest write-once run-anywhere language that lets me build a GUI while making distribution as simple as having the user install Java Runtime Environment 7 and running a single JAR file.

I have not worked out the data storage yet.  I am considering making it the application with the client-server model, where the user would interact with via browser.  It will not be a Java Applet as I typically discourage users from having Java available when web browsing, but it could be configured so one server allows multiple users to maintain their observations in a single database.  Views could then be generated based on this aggregated data.

For starters I am going small.  I have never done an open source project, so this is pushing me outside of my comfort zone and I am likely to make mistakes.  My initial goal will be to get a client side application to run on a host where the user can record and display their observations in the manner described above.  I accept all criticism or suggestions.

Base build is taking place in an Ubuntu 12.04×32 VM using Eclipse 4.2 and connected to github. (https://github.com/VernMcC/ArtifactIR)  There is nothing there yet, just the project shell, but I will be committing to this location as I do work.  The JRE used for the build will be JRE7u7.

Threat Analysis:  The best threat feed is now Twitter.  There are a couple of users that I follow that pop up in a special feed.  I find my response time to emerging threats has been improved as a result. At the top of this list is Kimberly ‏@StopMalvertisin, SophosLabs ‏@SophosLabs, and 0xǝrror ‏@0xerror.  These seem to bring to light threats that I find myself responding to at the time or find I need to start paying attention to. If anyone has others that are good to follow for emerging threats, please add them to the comments.  (There is also a collection of other security professions to follow if you enjoy security humor.  I won’t call them out here as I am afraid that might be seen as offensive.)

Incident Response: Knowing how to write non-production code to parse data or automate tasks makes the job a lot easier.  Just like in Malware Analysis, being able to do some rudimentary code (things the coding purists would gasp at) can dramatically increase the efficiency of a response.  I still enjoy Java (yes I use Java), Autohotkey, and Bash Scripting as my go-to languages, but am slowly picking up Python and Ruby.  I rarely use any of the MS sharp languages or C/C++.  An important thing to learn about any applications you use, to collect or store data, is what the API looks like or what the data schema is.  This makes writing code to access the data or parse it a lot easier and more accurate.

And if anyone missed it, Alex Lanstein from FireEye had a good in SC Magazine.  I particularly enjoyed his quote: “…whether an attack is truly APT or simply a well-financed adversary, the infiltration and exfiltration techniques are nearly identical.”  Link: http://www.scmagazine.com/fact-or-fiction-dissecting-the-myths-of-advanced-persistent-threats/article/247941/

Video Here: http://channel9.msdn.com/Events/TechEd/NorthAmerica/2012/SIA302

I was in the middle of an intense analysis session of some complex malware when I was notified of a brand new piece of malware that needed to be analyzed immediately.  As I went through the process of acquiring it, my boss approached, pulled up a chair, and asked how long before we knew anything.  On the earlier case, I had just reverted my lab VM to a validated base to some kernel level debugging of the samples.  I had only moved in the samples that I needed, but had not begun the analysis. No other activity had taken place in this lab VM.  Instead of taking the time to snapshot the lab’s current state, revert to my base, and analyze the new sample in a clean and validated environment, I just moved the new sample in.  I intended to get some quick static analysis done to decide how much time and what priority this new sample would need.  Then but for one click I could have been a hero.

I started my analysis by grabbing PE info and sending the sample into a hex editor.  This is accomplished with a right click context menu item.  I right clicked on the wrong sample and instantly started seeing similarities to the earlier case which I had dozens of hours in already.  I announced this linkage and that false intelligence lived for about 90 seconds. The skeptic in me realized the presence of the previous case’s sample in my analysis folder had to be excluded.  I quickly noted my mistake and announced it, but some damage had already been done.  No big bells or whistles had gone off, but our unit had already begun to emotionally ramp of for the response needed if this new malware was connected to the case I was working on.

After getting the genie back in the bottle, I built the lab correctly.  I blew away the lab VM, reverted to a validated clean base, moved in the new sample, and within 10 minutes had a decent  analysis done.  It was enough to let the rest of the team start hunting and to identify the malware as being a polymorphic variant of a known malware family.  That made further analysis beyond indicator collection low priority and I went back to my earlier case.

So to save the 5 minutes (max) that it would have taken to take a snapshot and then revert to the validated base image of the lab, I risked spinning our response unit up to full throttle in error and possibly harming our image with executives due to generating a false lead.  That is definitely not worth 5 minutes.

For those that don’t get the title: http://www.youtube.com/watch?v=jyaLZHiJJnE

Often when discussing malware, analysts will speak mechanically about the observations and capabilities of a sample.  (How is it packed?  How does it maintain persistence? What are the artifacts that let me find it on other machines or detect it on the network?…)  But a lot of what we need to do involves gathering and controlling information.  That is more like intelligence and counterintelligence operations and can be as hard as reverse engineering the sample itself.

Where did we get the sample from?  Who knows that we have the sample?  Who are we willing to share the sample with?  Have we seen similar samples that can be grouped or linked to another investigation? What is the sample trying to do (cross over from the mechanical approach)?  Does a sample demonstrate intelligence of the target or was it a dummy weapon that just happened to compromise a host?

When we are conducting our investigation and need to acquire a sample, the sample may be obtained from a host that we control.  These samples may not be droppers or installers, but we can gather a lot of information from the persistent samples left on a host.  Other times we will need to reach out in order to obtain a sample or the original dropper.  We may be examining links in an e-mail, document, or web history; we may find URLs for payloads in a sample we are analyzing; or we may just be doing research on a family of malware.  Some sources are safer than others, like when we acquire from a vetted malware dump or malware tracking site.  Others may be a compromised site that has the malware and some may be sites controlled by an adversary.

When we reach out for samples, do we fully consider the potential intelligence we are exposing? Some of the questions we need to ask ourselves include:

• What IP address does our traffic appear to come from?
• Who has visibility of the last leg of our route?
• What does a service provider know about their customer?
• How normal will the generated traffic appear?

Things like TOR, anonymizers, and virtual private servers (VPS) in the cloud can be used to obscure the source of traffic.  Entrance and exit nodes from TOR, an anonymizer service, or a VPS provider, have some degree of visibility of our traffic.  Even your local ISP and whatever DNS service that we use have some degree of visibility of our traffic.  Risking exposure of this intelligence is something that we want to minimize, but it is something we need to weigh against the goals of the investigation.  If the investigative goal is focused on short term response objectives due to exigent circumstances; risking exposure of the intelligence while consuming services may be warranted.  If the investigative goal is to not tip off an adversary as to what an organization is investigating, then a more methodical and discreet approach would be called for. If an investigation is in response to a well-known family of malware or is part of a highly visible malware campaign, exposure may not be as risky.  If it appears to be an unknown or custom malware sample or is part of an obviously targeted attack, limiting information exposure may be the most important investigative goal.

When selecting service providers (free or paid), we need to ask questions that include:

• What is the provider’s reputation?
• How strong is their security?
• Do they have more important customers?
• How do they make money or why are they offering the service?
• What do they do with information they gather?
• Are they bigger than you?
• What information do they reveal about their consumers?
• What information do they reveal about the services rendered?

Reputation can go a long way but the other questions still need to be examined to judge whether the reputation is sufficient to warrant use of a service.  If a provider has been around for a long time, provides quality service, demonstrates strong security of their consumer information, has clear and open policies, and has a clear purpose for offering the service, then consuming the service is likely low risk.  If a service provider exposes submissions for public examination or other customer examination, use of even the most reputable and useful provider needs to be weighed against the guarantee that the information submitted will be exposed. This includes Anti-Virus (AV) product providers.  Useful information could be revealed to an adversary if the AV product provider automatically includes all submissions in their signature updates to their other customers.

Some of the other issues we must address are how signals may be received.  What if the malware we are investigating is targeted at a limited number of potential victims or a single potential victim?  If we begin to generate traffic with sites under the control or observation of our adversary, then we may be tipping our hand that we are on to them.  If we discover a malware sample and reach out for help, either to other organizations or to service providers, we run the risk that the information that we have the sample will be exposed.  For example: if we submit all of our discovered malware to a cloud based AV scanning service for triage, our adversary could just set up an automated query for the malware hash value and determine when the malware has been discovered.  This would allow our adversary valuable time to reconfigure the malware or take an offensive act, early in our investigation, that we could otherwise have avoided had we not allowed the information to be exposed in a cloud service.  This is where we have to use judgment in determining whether the information that we gain outweighs the potential increase in risk due to exposing the information.

One of the biggest times that we can release information is when we start cleaning malware off of our systems.  When our adversaries see their number of compromised hosts dropping or appear to lose communication with all of them at once, they may attempt to take counter action.  This is why it is important to control when this signal is sent (it cannot be avoided when it is time to clean).  If the incident response investigation is complete, the malware has been analyzed, all compromised hosts have been identified, all of the stake holders are in place, and a remediation action plan is ready to be deployed, the adversary will be at a significant disadvantage compared to when they get this signal earlier and can counter during the early phases of the response process.

Last, we have simple information control.  Last year the story of the RSA breach piqued my interest as an intelligence and counterintelligence story.  It appears that early in the response phase, the malicious attachment that resulted in the adversary gaining a foothold was submitted to a cloud service for scanning.  The loss of control of this information later led to some embarrassment because RSA was no longer able to control and limit access to that attachment.  I am not questioning the judgment call because I do not have all the facts that were available to the decision makers at the time.  But the story did bring to light the need to consider these longer term information control issues.  It is also possible that the adversary became aware of the detection based on the publicly available information that was exposed at the time the sample was submitted to the cloud based service.

Some other ways we may inadvertently give away intelligence and some things to consider as counters to the risk:

• A unique or rare browser footprint may be associated with a certain research group or individual.  This would permit an adversary to know when that research effort is examining them. Panopticlick (https://panopticlick.eff.org ) is a good way to check whether your total browser footprint is common or rare.  Take steps to make your browser more common.  Use a live CD of a vanilla Linux distribution to do your research browsing.
• If an adversary has released malware to a limited number of targets or a single target, a user agent string not identical to the malware may alert them they are being examined and by whom.  Full malware analysis in a test network, with something like INETSim, some of the fake tools in the REMnux distro, or FakeNet (Full disclosure I have not had time to test FakeNet yet) can provide this information so exactly matching requests can be generated if needed.
• Over time, using the same IP address to do research may result in more advanced adversaries identifying traffic sources.  Using TOR, an anonymizer, or a consumer ISP with DHCP can help counter this.
• Service providers may be information gathering on us.  If we submit 20 or more polymorphic variants from the same malware family, it would be reasonable for the service provider to assume we are responding to an overrun of our network by that malware family.  Could that information alone have commercial use? I have not formulated a defense against this other than increasing the weight given to not submitting.
• What does our DNS traffic reveal? Using and rotating DNS providers can limit this information, but unless you have the money to support your own DNS root, you just have to realize the information you expose.
• Even if we don’t touch the adversary’s hosts, does our Internet research give away information about what we are responding do? Did we remember to turn off prefetching in Firefox before we started to conduct research in Google?  Easy solution, turn off prefetching and any other function that tries to cache potential next stops while browsing.

It is important for a malware analyst, and other analysts responding to an incident involving malware, to identify when some action they take could expose sensitive information.  When dealing with scatter shot malware that is part of a publicly discussed spam campaign or otherwise high profile/public incident, the potential cost due to the risk of releasing information is generally low.  But when faced with an unknown sample, or something with limited exposure, the cost due to exposure rises.  These decisions are something I have beaten myself up over, vowed to learn from, and then committed to making better decisions the next time.

The goal of a PXE Boot Server is to enable a disk image to be loaded onto a target machine when it boots.  The target machine is configured to look for a PXE server at boot time.  Essentially, the target machine looks for a DHCP server that will assign it an IP Address on the LAN and then load a small executable onto the target machine.  This executable is a boot time loader.  This small application can do many things, including uploading the current disk image to the server, download a previous disk image and write it to the HD, or wipe the HD.  The machine may also be booted from the HD normally, without any of the boot loader’s functions being called on. The easiest free method to do this is to build a FOG Server based on the FOG Project, http://sourceforge.net/projects/freeghost/, by Chuck Superski and Jian Zhang.

In a Malware Lab, a PXE Server is useful for making multiple images of a target machine in various configurations and being able to select the configuration initialized when the target machine is booted.  This allows for a target machine to be built with various tools installed and pre-configured on a variety of OS versions.  For example, I often use WinXP SP3 with a variety of Java, Flash, and Adobe Reader installed when trying to identity what a sample is exploiting.  I may also want to run a sample or view a page in IE6, IE7, IE8, or IE9. In large organizations that have multiple baseline images for deployed hardware, a copy of each baseline can be maintained. I can build the hardware up the way I want it, with the installed applications pre-configured, and take an image of its state.  This is much like when using VMware Workstation or another hypervisor application that allows a user to take a snapshot of a target machine’s state.  This image can then be used to restore the target machine to the state it was in when the image was taken.  While this is nowhere near as fast as hypervisor snapshotting, it is fast enough to be useful.  Typically I experience times between 5 and 20 minutes to restore a machine based on whether I have chosen to write the whole disk or simply partitions.

When building a Lab Network for use with a FOG Server, I isolate the network with a Linux server acting as a gateway offering a variety of services such as NAT routing, DNS, NTP, firewall, and NFS services.  I typically build these gateway servers with the Zentyal suit (http://www.zentyal.com/) which lets me manage the server with a web GUI and have granular control of the services available and access in and out of the network segment.  I store the drive images on a network share on this server, and leave the FOG Server just running DHCP and PXE boot services for this network segment.  I did not include the network storage in the example presented here.

For this demonstration I will be using VMware Workstation 8 with a custom network.  In reality, it may be better to utilize hardware for the FOG Server, but I have had success running it in ESXi and VMware workstation environments.

The custom network settings look like the following:

On this network there will be two primary machines:

1. FOG Server built with the VM image (v0.27), default Ubuntu install, and updated (Ubuntu first, FOG service second) with an IP Address: 172.16.25.2/24  GW/DNS 172.16.25.1
2. FOGTestXP (I could not get the imaging to work with VMware workstation target [It would image but not deploy correctly] so I will have to describe how I used it to image an old netbook.  The screen shots were created using the VMware FOGTestXP) A base install of WinXP SP3 with No updates, No AV installed, 1GB Memory, 20GB HD, and an IP Address of the OS 172.16.25.3/24

The first step is to open the FOG Server VM image for v0.27 in VMware Workstation.  For this example, I chose the normal Ubuntu install (defaults).  When the VM powers up, the install process begins automatically.  The user will be presented with a series of questions.  For my Malware Lab, these are the settings I used:

1. My answers to the install questions:
1. What type of installation would you like to do? [N] n
2. What is the IP address to be used by this FOG Server? [current address]172.16.25.2
3. Would you like to setup a router address for the DHCP server? [Y/n] n
4. Would you like to setup a DNS address for the DHCP server and client boot image? [Y/n] n
5. Would you like to change the default network interface from eth0? If you are not sure, select No. [y/N] n
6. Would you like to use the FOG server for dhcp service? [Y/n] Y
2. When prompted to log in, the username is root and the password is password
3. Once you get to the main menu, update Ubuntu first, then update FOG
   
1. The FOG Server needs access to the Internet to perform both of the update functions
2. When the FOG service is updating, the above listed questions will be asked again.  Select the same answers.
1. There will be an additional question about languages
2. This will update to the current version of FOG (as of this writing, 0.32)
3. Test the FOG Server by putting the FOG Server’s IP address into a browser on a machine that has a route to the FOG Server
1. Default username is fog and default password is password
2. You should be prompted to reset the database the first time you do this, just click the button and advance to the management GUI

Do a clean install of the target machine and then install any extra software that should be available when the image is restored. After the target machine OS and software are in the state that needs to be saved, gracefully shut down the target machine.  Make sure the target machine is connected to the same network segment as the FOG Server.  Boot the target machine and enter the Bios. Once in the Bios, insure that the network card is listed before the HD on the boot order list and save the configuration.

After saving and going through its reboot cycle, the FOG Server should answer the PXE DHCP request of the target machine and send the small boot loader application.  The screen on the target machine should look like this:
As indicated in the figure, select Perform Full Host Registration and Inventory and hit Enter.  Go with the default values for all questions. Do not image the machine during this phase.  After the Inventory is done, power off the target machine.

Back in the web GUI, click on the Image Management (Picture) icon in the ribbon:

Then click on New Image:

Enter a name for the image, select the default storage group, select Single Partition for the Image Type, and then click the Add button. 

Click on the Host Management (Computer) icon in the ribbon:

Click on List All Hosts:

Click on the Edit icon:

Select the image you made in the last step from the Host Image drop down box and click the Update button:

Then click on Basic Tasks in the side menu list:

Click on the Upload icon:

I usually check the Shutdown after task completion button, but it is not required.  Then click the Upload Image button.  This will set the task so the next time the target machine boots, it will send an image of the disk to the FOG Server.

You should get the response that the Task Started.

Then boot the target machine and it will automatically start the imaging process and when it is done, it will shutdown.

While there was an initial burst, my LAN settled on about 900 MiB/min so the image took about 4 minutes to save.

Once the image is saved, the target machine will shut down.  In the future, I can restore the machine to the state it was in when this image was taken by selecting it  for deployment in web GUI.  Get back to the basic tasks for the target machine, and selecting Deploy:

Click Image all Computers:

You should see a similar confirmation Screen:

And when you boot the target machine you will see it being imaged during boot:

There you have a basic setup for a FOG server.  While I prefer to use virtual machines, a FOG Server is a useful tool for analyzing malware that is VM aware.  It allows for the recording of a machine state and convenient GUI based management of the hosts and images.

When building a virtual network in VMware’s Workstation, I don’t want the host to have a virtual network connection to the guest that I use to test. This is a security issue as a sample could contain worm-like functionality that could potentially exploit this connection to break out of the test environment and compromise the host.I accomplish this isolation by building a custom network with the VMware Virtual Network Editor:

I just hijack what is normally the host only network in a default VMware Workstation install and modify it.
Make sure the check box for “Connect a host virtual adapter to this network” is unchecked.

This will prevent the host from connecting to the virtual network (note, when it is connected, the host is usually given the IP address of the network with the last octet being assigned 1 so 192.168.111.1 would be the host on this network).  Note that I leave the host running DHCP. (DHCP will appear to be served by a responder spoofed by VMware with the last octet of 254, so you may traffic from this spoofed address on your isolated network when machines make DHCP requests).  This is a function of the VMware application and not a true virtual network connection.  I then put a REMnux VM appliance in this isolated virtual network and connect.  So it ends up looking like this: 
Now, I have less worry about a sample being able to compromise the host.  It could still have an exploit for VMware’s Workstation product, but I don’t have to worry about a worm exploiting something that the host is vulnerable to across the host only network. The host is just not connected to the same virtual network as the guest.

To move files in or out of the machines on this network, I will move the REMnux appliance off of this virtual network and onto the bridged network so that it can access my full network. (Executing the ‘renew-dhcp’ in REMnux).  I can then use PSCP on the guest to pull files in from the REMnux appliance once it is returned to this isolated network.

I will also admit to cheating on this many times by just enabling copy a password protected zip file with the malware directly from the host and pasting it into the test machine.  This does require VMware tools running on the guest.

I guess I am less worried about VMware Workstation being vulnerable than I am the network connection to the host.

Ultimately, I get to snapshot and analyze the sample in the guest machine, using the REMnux appliance for network sniffing and service spoofing.

The most valuable tool for any malware analyst is the software that hosts their virtual machines in both their test and analysis environments.  While windows licenses cost money, the virtual machine software can be free.  Oracle’s VirtualBox, the QEMU Project, and VMware’s Player are free applications that can host virtual machines.  While I prefer VirtualBox due to the GUI combined with snapshot capabilities, the others provide some additional benefits.  Most of the malware that I have analyzed that was VM aware looked for VMware signatures.  A couple of the samples did things with the floating point processor to detect abnormal results and detected VirtualBox, but I have not observed any that directly look for QEMU signatures.   Additionally, if you have a server in your arsenal, you can install the free VMware ESXi hypervisor to build intricate networks and manage the multiple virtual machines through one interface.  One of the negatives of the ESXi application is getting memory snapshots from the hypervisor to the machine you use to do memory analysis.

For paid applications, nothing beats VMware Workstation 8.  It has the snapshot capabilities, virtual network editor, and can run VMs off remote hosts or act as a client for an ESXi server.

All of the applications can run in a Linux environment, which adds a layer of protection when conducting malware analysis.  Typically the malware to be analyzed will be targeted at Windows.  If the host has a Windows operating system moving malware around can pose a danger to the host or cause the host’s AV solution to clean a sample before you can get it the guest.  These problems can be mitigated by using password protected archive files to move malware around or utilizing scp to move files from non-Windows source to the destination.  But, with many of the desktop emulators, and the ESXi host, it is possible to move files through the clipboard or drag and drop with the host.  This can be particularly useful when exporting reports generated by tools in the test or analysis environment.

My preferred method is to use Workstation 8 on a Linux host. You can build a series of VMs that can be brought into service very quickly.  Tests can be run and then the guest reverted to a pristine state.  At least one of the guests needs to be a vanilla install of each operating system without VMware tools being installed.  VM aware malware can still detect that it is running in a VM, but the installation of VMware tools makes that check trivial since there are registry entries associated with them.  Keeping some vanilla installs without VMware tools lets you quickly move a sample into an environment where samples may otherwise miss they are running in a VM.

Last, no matter how many VMs a malware analyst has at their disposal, if they need to be fully covered for dynamic testing, they have to have some hardware they can quickly build up and then revert.  The easiest way to do this is with a FOG server and an old desktop or laptop.  Using the FOG server you can build a variety of hardware configurations, save the image, then select which you want pushed to the hardware at boot using the PXE boot options.  This is a lot slower than VM software utilizing snapshots, but it is a lot quicker than having to reconfigure a machine for a test each time.  I will plan on covering how to set up and deploy a Fog server in a later blog post.