Multiple SYSLOG Receivers with a Cisco NAC Appliance Manager (CAM)

According to Cisco’s documentation on configuring syslog on a CAM, you can only forward the NAC logs to a single external log server. If you’re willing to get down and dirty with the Linux operating system underneath, then this document will show you that this is simply not the case.

To get started, tweak the default logging settings within the NAC web interface, this screen-shot shows I’m sending the syslog to the local host as local6 messages, this change will send a copy of the “normal” NAC event logs to the localhost syslog server.

Next we need to enable the localhost syslog server; the CAM is build upon a Fedora image, so the SYSLOG daemon is already running it’s just not listening on UDP 514 (thus not yet receiving the logs configured above). Change /etc/sysconfig/syslog , the line:

Now that the local daemon is recieving the files we need to change /etc/syslog.conf, here we will make two changes, One: we will write a copy of the NAC events to disk – this will allow us to see what events the “NAC application” is sending. The second change we’ll make is the forwarding configuration, we will put in two lines (for both our syslog hosts) so that we send forward the syslogs to two different servers – which was our original intention :)
Add the following lines to /etc/syslog.conf :

# Log Messages sent from Cisco NAC Application to dedicated File
Local6.*	/var/log/CiscoNAC.log

# Forward all syslog messages to host1
*.* 	@loghost1
# Forward all syslog messages to host2
*.* 	@loghost2

*NOTE: loghost1 & loghost2 need to be resolvable via DNS or in /etc/hosts !!

Finally restart the syslog daemon /etc/init.d/syslog restart

It’s good practice once we’ve made changes to clear up after ourselves, these are some option steps you can take.

Add /var/log/CiscoNAC.log to logrotate, so that it doesn’t just grow and grow until you run out of disk space. This is done by editing /etc/logrotate.d/syslog before /var/log/messages insert /var/log/CiscoNAC.log

You may also want to compress your syslogs, edit /etc/logrotate.conf and uncomment the word compress (remove the “#”) .

Important Note
When performing NAC upgrades, Cisco provide operating system package upgrades & changes, it’s important to check that after an upgrade this config changes still exist, also I take no responsibility for Cisco’s TAC not wanting to support you because of the changes made!

MARS: Zone product or package version does not match

I’ve been having problems getting my Cisco MARS Local and Global controllers to synchronise their topologies. This error message vexed me for a few days, but thankfully Cisco’s TAC solved it for me.

If you read Ciscos troubleshooting guides they will tell you to check that the MARS Local & Global controllers are running the same version, and to check that the SSL certificates are copied/pasted correctly.

If after checking the above Cisco recommendations and the additional basics ( network connectivity / ntp / timezones etc) check that both MARS boxes are running and have downloaded the same version of IPS signatures; under Admin -> IPS Signature Dynamic Update Settings -> Update Now.

It fixed the problem for me!

CS-Mars V6.0 in VMWARE (Franken Mars)

Emulating software is a very grey area for Cisco, they make their money by selling boxes so I guess officially Cisco don’t approve of things like GNS3 and PEMU. BUT cisco make a lot of their money from techies training in Cisco products who then get their management to buy boxes their certified in, as a result cisco appear to turn a blind eye to emulating their products for personal training purposes :)

So, I’m installing a CS-Mars box in the next couple of weeks and wanted to know what’s new in version 6. How to setup version 4 is already document here in this franken cs-mars guide, the thing is to upgrade from 4 to 6 is a re-image of the box. Upon re-imaging my VMWare appliance I realised that the lilo commands linux rw init=/bin/bash didn’t appear to work anymore. As a result I have a v6 mars box I can’t use due to a licensing problem.

To get this working read through both the old instructions, and what I have written.

The init/boot sequence of a mars box looks very much like a centos/fedora boot, so I thought up a cunning new plan. I downloaded the 1st installation CD of centos 5, after booting this CD instead of hitting “enter” and running the anaconda installer I typed linux rescue, this boots my appliance into a root linux shell. (See Update Below, boot from CentOS straight after MARS installs, don’t let MARS boot!)

What happened next was a little hit and miss, if you’re lucky you can type

mkdir /mnt/opt
mount /dev/md2 /mnt/opt

you can then

cd /mnt/opt/janus/release/bin
mv pnlicense
echo "/bin/echo d84f7ceaf50f9c45683e2efb77752d4f:License verified:4:0:0:4" > pnlicense
chmod +x pnlicense

as per the old documentation.

If you’re unlucky this “mount” will fail, in this case ls /mnt/sysimage if you can’t see any files issue mount /dev/md1 /mnt/sysimage otherwise the plan is to change the root password so that we can edit the pnlicense file later.

Using vi edit /mnt/sysimage/etc/passwd, and change…




Next, setup your editor variable, and edit the suders file…

EDITOR=/mnt/sysimage/bin/vi;export EDITOR
visudo -f /mnt/sysimage/etc/suders

and add..

pnadmin ALL=(ALL)       NOPASSWD: ALL

Reboot by exiting the shell.

After the reboot login as pnadmin, you should now get a standard linux bash shell rather than the “hardened” cisco one. Change the root password…

sudo su
passwd root

And put /etc/password back to how it was. Now from the “pn shell” you can type expert and your root password will work and you’ll have root access to your mars box. With you new root access you can change the pnlicense file as described before and complete the setup process. :cool:

UPDATE: As commented by secopt below, to make this work you need to boot from the CentOS disk straight after the MARS image as installed, if you let the MARS OS boot (and start doing the oracle thing) then for some reason the mount commands don’t work!

UPDATE2: The mount command doesn’t work if you let MARS boot the 1st time as it changes the superblock, rokov has posted the following work around below…

  1. Assemble RAID
    mdadm –assemble /dev/md0 /dev/hda3 /dev/hdc3
  2. Change ext3 superblock magick number
    dd if=/dev/md0 skip=2 count=1 | sed ’s/\x5A\x7B/\x53\xEF/’ | dd of=/dev/md0 seek=2 count=1
  3. Mount partition
    mount /dev/md0 /mnt
  4. Do anything you want with it.
  5. Unmount partition and change magic back
    umount /mnt && d if=/dev/md0 skip=2 count=1 | sed ’s/\x53\xEF/\x5A\x7B/’ | dd of=/dev/md0 seek=2 count=1

Strange ASA ARP Replying Behavior

I’ve been implementing a few Cisco ASA’s recently, and I blogged about this strange behavior; well I came across another one yesterday.

Take a look at this debug arp….

CiscoASA# debug arp
debug arp  enabled at level 1
CiscoASA# arp-set: added arp outside 001e.7000.1234 and updating NPs at 4301321940
arp-set: added arp inside 001a.7100.1234 and updating NPs at 4301321940
arp-in: request at outside from 001a.3000.1234 for 001e.7a51.1234 arp-in: rqst for me from for, on outside arp-set: added arp outside 001a.3000.1234 and updating NPs at 4301326660 arp-in: generating reply from 001e.7a51.1234 to 001a.3000.1234
arp-in: request at outside from 001a.3000.1234 for 001e.7a51.1234 arp-in: rqst for me from for, on outside arp-set: added arp outside 001a.3000.1234 and updating NPs at 4301326660 arp-in: generating reply from 001e.7a51.1234 to 001a.3000.1234 arp-in: request at outside from 001a.3000.1234 for 001e.7a51.1234
arp-in: rqst for me from for, on outside arp-set: added arp outside 001a.3000.1234 and updating NPs at 4301326660 arp-in: generating reply from 001e.7a51.1234 to 001a.3000.1234
arp-in: request at outside from 001a.3000.1234 for 001e.7a51.1234 arp-in: rqst for me from for, on outside arp-set: added arp outside 001a.3000.1234 and updating NPs at 4301326660 arp-in: generating reply from 001e.7a51.1234 to 001a.3000.1234 arp-in: response at outside from 001a.3000.1234 for ffff.ffff.ffff arp-in: updating gratuitous ARP - 001a.3000.1234 arp-set: added arp outside 001a.3000.1234 and updating NPs at 4301326660 CiscoASA#

The firewall is replying to arp requests even though both the source & destination of the traffic are on the same (outside) interface, now I haven’t manged to work out why the firewall was doing this, but I did find a fix on the cisco forums.

sysopt noproxyarp outside

Names, IPs & MAC’s have been changed to protect the innocent.

Cisco NAC SSO Port List

Note to self, the ports I need to allow thru the Un-Authenticated ACL for Active Directory SSO to work…

TCP 88,135,389,636,445,1025,1026 				
UDP 88,389,636 


Cisco ASA and 7905 IP Phone Weirdness

I came accross something odd the other day, I had some Cisco IP Phones on a DMZ interface and the Call Manager was behind the inside interface. If you made a call from a 7940 to a 7940 everything worked fine, if you made a call from a 7905 to a 7940 it failled!

I ran a packet capture and found that the phone was “bouncing” the RTP stream off the firewall rather than connecting directly to the peer phone… very weird! The problem was solved by enabling…

same-security-traffic permit intra-interface

I thought I post this for some future googlers!

Backup Interface on Cisco ASA Firewall

I tweeted a little while ago about Nokia recently supporting interface failover within IPSO, well it looks like Cisco’s ASA Version 8 software can do it now too!

The following example creates two redundant interfaces:

asa(config)# interface redundant 1
asa(config-if)# member-interface gigabitethernet 0/0
asa(config-if)# member-interface gigabitethernet 0/1
asa(config-if)# interface redundant 2
asa(config-if)# member-interface gigabitethernet 0/2
asa(config-if)# member-interface gigabitethernet 0/3

Reference: Adding a Redundant Interface

Cisco VPN 3k Config for iPhone

Recently I was asked if I could help setup a VPN connection between an Apple iPhone and a Cisco VPN Concentrator 3000, my 1st round of googling didn’t look good, there’s a discussion here complaining about how crap vpn support on the iphone is; further searching lead me to a Cisco document which specifically targets mac clients, this document is for ASA configuration, but if you look carefully* everything you need is in there.

*No, I didn’t get this working 1st time, it took me a good couple of hours of googling, but looking back I can see that all the info is there.

The key to getting this working is that the iphone side is not as configurable as it should be, so if you’re trying to get this to work you need to be talking to the IT administrator to get the concentrator side changed. The 1st word of warning is that the iphone client doesn’t support group authentication, so you’re going to be changing the base group, now by default most “production” groups will inherit settings from the base group, so you will need to make sure that if you change anything in the base group that it doesn’t effect your other L2L or Remote Access tunnels. (You have been warned.)

To get started, for whatever reason the iphone only supports cisco’s NAT-T implementation of IPSEC, so if you have a firewall or access-list in front of your concentrator you’re going to need to open up UDP 4500, then enable NAT Transparency. Another word of warning about NAT-T, we found that existing VPNS to Cisco Routers started to fail after enabling this, which was a bit odd as NAT-T wasn’t enabled under any of the L2L profiles, anyway, to fix the issue we enabled NAT-T on the routers (again make sure UDP 4500 is allowed though any ACLs) and under “conf t” issue:

crypto ipsec nat-transparency udp-encapsulation

So, back to the cVPN3k config……

Configuration -> Tunnel & Security -> NAT Transparency

So a quick explanation of the above so you get the idea; from the tree on the left, click “Configuration” then “Tunnel & Security” then “NAT Transparency” and tick the box next to NAT-T.

Now you need to setup your PHASE 1 Proposal…

Config -> Tunnel & Sec -> IPSEC -> IKE Proposal

I called mine iphone, and you need to configure the following settings.

  • Authentication: Preshared Key (NOT the one with Xauth)
  • Hash: SHA-1
  • Encryption: 3DES
  • DiffeHelmen: Group 2

After phase one, comes PHASE 2:

Config -> Policy Manage -> Traffic Mgnt -> SA

Again, add the following settings and I called mine: iphone

  • Authentication: ESP / SHA
  • Encryption: 3DES
  • Enacapsulation: Transport
  • IKE Proposal = iphone (or whatever your phase 1 was called)

Then finally we start working with the groups, so as mentioned above you need to work with the base group:

Config -> User Management -> Base Group

And you need to enable the following, the other settings will be optional:

On the Base Group Tab,

  • Tunnel Protocol: Tick “L2TP over IPSec”

On the IPSEC Tab,

  • Authentication: Internal or NT depending on what you’ve already configured for other Remote Access Profiles.
  • IPSEC SA is set to: iphone
  • Default Preshared Key: Set this to something really really long (this will be your secret on the iphone)

On the PPTP/L2TP Tab,

  • L2TP Authentication Protocols: Tick MSCHAPv1 / MSCHAPv2
  • L2TP Encryption: Tick 40 & 128 B

DONE! Now with a little bit of luck your iphone should connect.

A Quick note about comments: All support requested will be deleted, I don’t have access to a concentrator to offer any meaningful advise, you use the above config at your own risk.

PKI: Cisco Routers as a Certificate Authority

I’ve had a new request in recently, as part of a move to SCEP + Certificates (away from pre-shared keys) a customer has asked if we could use the PKI CA build into Cisco’s router IOS. Now is this is a new idea to me; in the past people have either “plumped” for Microsofts CA implementation or cooked something up themselves with openssl.

Cisco’s IOS Security Guide (you may need a CCO Login) clearly states that it’s possible and that it supports SCEP auto-enrolment, so I thought I’d give it a go!

I don’t have any routers available at the moment, but I’ve been able to get things tested within gns3 ( gns3 screen shot). What I’ve done is setup a basic implementation, I have a router called “CA” which would be the root certificate authority, which would typically be on the inside of your network; then I have a router called “VPN_A“, this router would be the HQ VPN Termination device, it would be where all the remote (or branch) routers connect to, this router is configured with SCEP and is able to HTTP download the CRL from CA. The final router is “VPN_B” this router has been enrolled manually with a “copy / paste” and does not have HTTP Access to the CA, below is a screen shot of what I’ve done.

Example configs and a .net config file, which can be imported into gns3 (with a little tweaking) are available in my cisco directory, hopefully these examples give enough detail on how to manually or automatically enroll the routers and setup and IPSEC VPN, obviously they’re not complete configs (no usernames & pass’s set for a start) but there should be enough for someone to integrate this into their existing templates.

How to Import Vendor Specific Attributes into Cisco Secure ACS SE Applience

I wanted to write a document on how to import RADIUS VSA’s (vendor specific attributes) into cisco’s ACS SE (Solution Engine) appliance, the reason being that I couldn’t find any good examples on the net and cisco’s documentation just wasn’t clear enough.

My purpose was to use RADIUS authentication between a Nokia IPSO appliance such that users who access voyager or ssh get authenticated centrally; for RADIUS authentication to work your authentication server (in this case ACS) needs to supply the AAA client (in this can the ipso box) with a “return list attribute”. By default ACS doesn’t have the nokia attributes; to import attributes you need to get your hands on a dictionary file, for nokia ipso it’s /etc/nokia.dictionary – I’ve a copy here.

In you dictionary file you need to pick out some key elements, firstly the IANA-assigned enterprise code for the vendor and secondly a list of attributes to add. Using my nokia example the vendor code is the top line:

VENDOR Nokia 94

Thus the code is 94 , and everything below that are attributes.

So… Getting started with ACS, firstly if you have AAA clients which you want to use the new attributes you are going to need to delete them, and to be save reboot ACS. Now the import is done via the RDBMS sync process, since you do not have OS level access to ACS you need to upload a file called “accountActions.csv” (case sensitive), uploading this file tells the internal database to perform some commands or actions, examples would be to bulk import some users or bulk group changes, in our case we’re going to insert a new “Vendor” into the RADIUS database, and then insert some attributes.

I have created a file called createVendor_accountActions.csv if you renamed it to accountActions.csv and uploaded it to your ACS box via the RDBMS Sync tool (under system configuration) it’d perform the following actions:

    • Command -1
    • Priority – 8
    • Action – 350 (Create new Vendor)
    • Vendor Name – Nokia
    • ACS Vendor Number – Auto Assigned
    • Vendor ID – 94
    • Date of DB Transaction – 25/09/2007 13:00
    • Command – 2
    • Priotity – 0
    • Action – 355 (Restart ACS Services)
    • Date of DB Transaction – 25/09/2007 13:00

The command numbers are just like primary key fields in a database or row numbers in a spreadsheet, they need to be unique and incremental for each csv file, and the priority specify and order to apply the commands, you I guess you could set the priorities all to 0 and rely on the command number to process the file in order, but I set a priority just in case. After you apply the file ACS will be temporarily unavailable as the services restart.

Now, we look at one line of importAttributes_accountActions.csv, again it would need to be renamed to accountActions.csv, before uploaded, and lets take a look at one line.

  • Command -1
  • Priority – 7
  • Action – 352 (Add VSA)
  • Attribute Name – Nokia-IMSI
  • The vendor to assign the attribute to – 94 (Nokia)
  • Attribute ID – 224
  • Attribue type – integer ( can only be integer, string or ipaddr)
  • Date of DB Transaction – 25/09/2007 13:00

Hopefully this all starts to make sense when looking at your dictionary file, again the final line of the file restarts the services. An important thing to not here is that if you create a new vendor you need to re-start the services before you can apply an attribute to it, and you need to restart the services again to use the attributes… at this point here it’s probably worth mentioning that the version of ACS SE I’m using now (4.1) is a windows appliance, so if at any point your box hasn’t done what you think a reboot won’t hurt ;)

Now you can add your AAA client and in my example you could set the vendor to RADIUS (Nokia) , if you then go into interface configuration RADIUS (Nokia) will appear, go in there and tick all the boxes for “group”, finally if you go into your group setup at the very bottom will be a list of attributes you’ve imported and can use ! :cool:

Just in can you need them here are my references:

RDBMS Sync Import Definitions

Importing an accountActions.csv file into ACS SE

Universe CD version of RDBM SSync import Defs

OSPF & Cisco ASAs

One of the interesting things about ASA’s is the fact that it supports running two OSPF Processes. This was a great decision by cisco, if a business has two different OSPF domains the chances are they are owned by two separate parts of the business, so where would be a better place to put a firewall?

I’ve put together a basic lab / config to test out the functionality, obviously this doesn’t address IP conflicts which are quite likely to happen in a real world scenario, but you do get the general idea. In my cisco config directory you’ll find two router configs and an ASA config. Each router is intended to represent each ospf domain, the ASA will then re-distribute the routes into each process… Note: you’ll see some “show” commands at the end of the config files.

I actually put this together as a “just in case” type thing, but I expect this to come in very handy in the future ! :cool:

News – Fooling Cisco’s NAC network access control

Just Found this,

heise Security – News – Fooling Cisco’s NAC network access control

Security experts at the Black Hat conference in Amsterdam have demonstrated how Cisco’s NAC network access control can be fooled. In a live demonstration using a modified Trust Agent, Michael Thumann and Dror-John Rcher from ERNW were able to gain full access to an NAC protected network using a computer which did not comply with network policies.

Although it was obvious that hackers would target the the Trust Agent, it’s interesting to read a sucess story.