LINICKX.com

bash: ls modified or added files

Nick Bettison — Fri, 13 Mar 2015 08:00:00 +0000

One of the most useful features of OSX's finder is bing able to sort files by modified or added. In my typical workflow I use a $SHELL to move around directories and then use open . to open finder in the right location.

To speed things up a little more, I've knocked up a couple of alias's for my .bash_profile

lm

Works on both linux and OSX

    listmodifiedfiles() {

        lmdir="."
        lmdir=${1:-$lmdir}

        if [ -z "$2" ]
        then
            lmnum="-5"
        else
            lmnum=$(($2 * -1))
        fi

        ls -1t $lmdir | head $lmnum
    }
    alias lm=listmodifiedfiles

This adds an lm alias/command that will list the last 5 modified files. The command takes two options the first being a directory, the second being the number of files to list.

E.g.

    linickx:dotcom nick$ lm ./content/posts/ 3
    bash-ls-modified-or-added-files.md
    css-styling-nginx-directory-listings.md
    good-bye-wordpress-hello-pelican.md
    linickx:dotcom nick$

la

Works on OSX only

    listaddedfiles() {

        ladir="."
        ladir=${1:-$ladir}

        if [ -z "$2" ]
        then
            lanum="-5"
        else
            lanum=$(($2 * -1))
        fi

        ls -lrt -d -1 $ladir/* | grep -v '^\.$\|^\.\.$' | xargs -I {} mdls -name kMDItemFSName -name kMDItemDateAdded {} | sed 'N;s/\n//' | grep -v '(null)' | awk '{gsub(/"/, "", $7);print $3 " " $4 " " $7}' | sort -r | awk '{print $3}' | head $lanum
    }
    alias la=listaddedfiles

This add an la alias/command that will list the last 5 added files. As before, the command takes two options the first being a directory, the second being the number of files to list. e.g.

    linickx:~ nick$ la ~/Downloads/
    GZC4573714.zip
    AdobeFlashPlayerInstaller_17au_ltrosxd_aaa_aih.dmg
    Defaultselfsignedservercerti.pem
    linickx:~ nick$

NOTE This command takes a long-ish time to run on big directories as each file has to have it's meta data checked prior to output. If anyone a suggestion to make this quicker I'm all ears !

References:

http://stackoverflow.com/questions/15691359/how-can-i-list-ls-the-5-last-modified-files-in-a-directory
http://apple.stackexchange.com/questions/86307/can-i-list-files-ordered-by-date-added-to-a-folder-from-a-command-line-tool-like

OS X: Converting videos but keeping the timestamp

nick — Wed, 06 Mar 2013 18:24:00 +0000

I got a new computer, w00t! My family like to use the point-and-click camera for adhoc videos whilst we're out and about, the thing is it's very old and records in a rubbish AVI format which consumes massive amounts of disk space, b000!

After a bit of googling I wrote the below, it'll search through your disk and find AVI files, check that the file extension is .AVI and then convert it to .mp4; for bonus points it'll change the timestamp of the mp4 to match the avi so that it'll import into iPhoto albums nicely.

#!/bin.bash
export IFS=$'\n'
for i in $(find ./ -type f -name '*.AVI') 
    do 
        if [ ${i: -4} == ".AVI" ]
            then
            echo "Converting $i"
            ffmpeg -i "$i" -s 480x320 -aspect 4:3 -b 768k -ab 64k -ar 22050 -r 30000/1001 "${i%.AVI}.mp4"
            sleep 5
            TSTAMP=`gls -l --time-style=+%Y%m%d%H%m "$i" | awk '{print $6}'`
            touch -mt $TSTAMP "${i%.AVI}.mp4"
            sleep 5 
            rm -f "$i"
        fi
    done

Quick Note: gls is the GNU version of ls not the built in BSD-MAC version, you get it via homebrew (brew install coreutils)

Building a free Dynamic DNS client with rackspace Cloud

nick — Sun, 30 Oct 2011 21:46:00 +0000

As a cloud server customer you get access to rackspace's free DNS service.

When I fist saw this product I had an instance light-bulb moment, I could stop paying for a dynamic DNS service and build my own private one. As a broadband (DHCP) user I have a very basic requirement of needing to regularly update an A record so that I can find my pc :)

To bring my idea into fruition I began researching; I need a cli tool which I could run from cron on my linux box (to send the DNS update requests to rackspace). In my research I found rscurl, a cli tool to control cloud servers, as rackspace have a standard API for all their products I have been able to use rscurl to develop rsdns.

rsdns is a series of cli tools to adding/deleting/changing rackspace DNS records, as part of the tool development I have created a script called rsdns-dc.sh to run on my machine, below is a short how to:

How to get free dynamic dns from rackspace.

Pre-requisit: This is for linux/mac, if you want to do this on windows you'll need bash, curl, awk, sed & dig installed - google is your friend.

The instructions below assume that you (a) have a domain and (b) have already changed your NS records to point to dns1.stabletransit.com & dns2.stabletransit.com.

Download the latest rsdns from gitgub and unpack somewhere, I like ~/bin/rsdns.

Go to your rackspace management portal and grap your username & API key (It's under "Your Account" -> "API Access")

Create a config file for rsdns (~/.rsdns_config) with your settings.

#!/bin/bash
RSUSER=linickx  
RSAPIKEY=123456  
RSPATH=~/bin/rsdns/

You need your domain created on rackspace, you can either use rackspaces GUI to do this (It's under "Hosting" -> "Cloud Servers" -> "serverabc" -> "DNS" ) or you can use rsdns, like so.

#!/bin/bash
[LINICKX@SERVER rsdns]$./rsdns-domain.sh -d www.linickx.com -e spam@linickx.com

Once you have a domain setup you need an A record, this step was a deliberate design to avoid any rouge cron jobs from creating a million records, the dynamic client will only update an existing record - not create a new one.

To create the a record you going to need an IP address, it can be something random like the below, or you can use http://icanhazip.com to get your actual current IP. Again to create a record, you can use the rackspace GUI (It's under "Hosting" -> "Cloud Servers" -> "serverabc" -> "DNS" -> "yourdomain" > "Add") or you can use rsdns....

#!/bin/bash
[LINICKX@SERVER rsdns]$./rsdns-a.sh -n dynamichost.linickx.com -i 123.123.123.123 -t 3600

In the above the TTL is set to 1hr, this is so that DNS caches do not keep the record too long. That's all the pre-work done, now lets get your dynamic host setup!

The script to update your a record is rsdns-dc.sh, and you run it like this...

#!/bin/bash
[LINICKX@SERVER rsdns]$./rsdns-dc.sh -n dynamichost.linickx.com

Easy huh? The script uses icanhazip to get your current IP, it then update the A record with it.

I never switch off my router so I have create a created a cronjob to run that script every 2 hours, plus the 1hr TTL should mean that the record is roughly in sync with my IP without making unnecessary requests - You can run the script more often if you like, just stay under the limits --> according to the API guidelines you can make upto 25 changes per min / 2 per second.

I use redhat based linux systems, so I can simply drop the following file called rsdns-dc into /etc/cron.d/ with this...

* */2  * * *     linickx /home/linickx/bin/rsdns/rsdns-dc.sh -n dynamichost.linickx.com &>/dev/null

Now we are actually done! Free private Dynamic DNS on your own zone, what more could you want?

Cisco ASA Firewalls and IP Ranges in ACLS

nick — Fri, 29 Jul 2011 15:05:00 +0100

I've google'd and I cannot find a way of creating a firewall range style object in an ASA, you know the kind of thing whereby you want to allow IP addresses 192.168.1.10 thru 192.168.1.20 in an ACL.

In my frustration I have given up and created a shell script which converts a CSV into an ASA output, simply create a two column CSV with Col A containing your starting IP and Col B containing you end IP.

The script is a recursive loop so should support large outputs such as 10.1.2.10 to 10.2.1.20 howvere I'm not actually sure you'd want that in your firewall config but I wrote the computability for the fun it!

Have fun, click "more" below if you can't see the script!

#!/bin.bash

# Commas separated VAR....
IFS=","
while read name firstip lastip
# Loop around CSV
do

# Split up our first ip into it's octects
firstipfirstoctect=$(echo $firstip | awk -F "." '{print $1}')
firstipsecondoctect=$(echo $firstip | awk -F "." '{print $2}')
firstipthirdoctect=$(echo $firstip | awk -F "." '{print $3}')
firstipforthoctect=$(echo $firstip | awk -F "." '{print $4}')

# Split up our last IP into it's ocects
lastipfirstoctect=$(echo $lastip | awk -F "." '{print $1}')
lastipsecondoctect=$(echo $lastip | awk -F "." '{print $2}')
lastipthirdoctect=$(echo $lastip | awk -F "." '{print $3}')
lastipforthoctect=$(echo $lastip | awk -F "." '{print $4}')

    # Re-set BASH
    unset IFS

    # Echo out the object GROUP name
    echo "object-group network $name"

    # Loop through 1st Octect
    for a in `seq $firstipfirstoctect $lastipfirstoctect`;
    do
        # test to see if we need to print the whole range
        if [ $firstipfirstoctect -lt $lastipfirstoctect ]
        then
            firstipsecondoctectCOUNTER="0"
            lastipsecondoctectCOUNTER="255"
        fi

        # first IP might not be 1
        if [ $a -eq $firstipfirstoctect ]
        then
            firstipsecondoctectCOUNTER=$firstipsecondoctect
        fi

        # last IP might not be 255
        if [ $a -eq $lastipfirstoctect ]
        then
            lastipsecondoctectCOUNTER=$lastipsecondoctect
        fi

            # Loop through 2nd Octect
            for b in `seq $firstipsecondoctect $lastipsecondoctect`;
            do

                # Same tests as before except, next octect.
                if [ $firstipsecondoctect -lt $lastipsecondoctect ]
                then
                    firstipthirdoctectCOUNTER="0"
                    lastipthirdoctectCOUNTER="255"
                fi

                if [ $b -eq $firstipsecondoctect ]
                then
                    firstipthirdoctectCOUNTER=$firstipthirdoctect
                fi

                if [ $b -eq $lastipsecondoctect ]
                then
                    lastipthirdoctectCOUNTER=$lastipthirdoctect
                fi

                    # Loop through 3rd Octect
                    for c in `seq $firstipthirdoctectCOUNTER $lastipthirdoctectCOUNTER`;
                    do

                        # copy / paste / tweak
                        if [ $firstipthirdoctect -lt $lastipthirdoctect ]
                        then
                            firstipforthoctectCOUNTER="0"
                            lastipforthoctectCOUNTER="255"
                        fi

                        if [ $c -eq $firstipthirdoctect ]
                        then
                            firstipforthoctectCOUNTER=$firstipforthoctect
                        fi

                        if [ $c -eq $lastipthirdoctect ]
                        then
                            lastipforthoctectCOUNTER=$lastipforthoctect
                        fi

                            # final octect... echo result.
                            for d in `seq $firstipforthoctectCOUNTER $lastipforthoctectCOUNTER`;
                            do
                                echo " network-object $a.$b.$c.$d  255.255.255.255"
                            done

                    done
            done
    done

done<./FirewallRanges.csv

GTD in your Shell

nick — Tue, 26 Apr 2011 21:27:00 +0100

For the last few months I've felt better at managing my actions and tasks thanks to Steve Losh's T. T is a simple python script that allows you to manage a simple task list from your shell prompt.

I use my shell all the time to ping stuff, renice processes and you know, whatever (like in the image below).....

... having my todo list right there in front of me has been really helpful (The numbers in the square brackets!). Steve's website suggests a quick update to your shell profile to add your task list to your prompt but I found running a python script each time I do anything quite slow, especially since I have two lists.

Below is my bash .profile file, there you can see the rather OTT change(s) I've made to integrate my todo lists into my shell prompt. I have two lists (work & personal) the number of things to do shown in my shell prompt gets updated each time the lists change (thanks to md5).... and I hide my work list over the weekend ;)

#!/bin/bash
alias p='python ~/bin/t/t.py --task-dir ~/Dropbox/Tasks --list personal.tasks --delete-if-empty'
alias w='python ~/bin/t/t.py --task-dir ~/Dropbox/Tasks --list work.tasks --delete-if-empty'

function nick_PS1 {

        if [ -e "/Users/nick/Dropbox/Tasks/Personal.tasks" ]
        then
                CurrentPersonalMD5=`md5 ~/Dropbox/Tasks/Personal.tasks | awk '{print $4}' | cat `

                if [ -e "~/.tasks.personal.md5" ]
                then
                        LastPersonalMD5=`cat ~/.tasks.personal.md5`
                fi

                if [ "$CurrentPersonalMD5" == "$LastPersonalMD5" ]
                then
                        if [ -e "~/.tasks.personal.no" ]
                        then
                                NumOfPersonal=`cat ~/.tasks.personal.no`
                        fi
                else
                        NumOfPersonal=`p | wc -l | sed -e's/ *//'`
                        echo $NumOfPersonal > ~/.tasks.personal.no
                        echo $CurrentPersonalMD5 > ~/.tasks.personal.md5
                fi

                if [ $NumOfPersonal -ne 0 ]
                then
                        LEFT='['
                        RIGHT=']'
                        Personal="p:$NumOfPersonal"
                fi

        else
                NumOfPersonal=0
        fi

DayOfWeek=`date "+%u"`
if [ $DayOfWeek -lt 6 ]
then
        if [ -e "/Users/nick/Dropbox/Tasks/Work.tasks" ]
        then
                CurrentWorkMD5=`md5 ~/Dropbox/Tasks/Work.tasks | awk '{print $4}'`

                if [ -e "~/.tasks.work.md5" ]
        then
                LastWorkMD5=`cat ~/.tasks.work.md5`
        fi

                if [ "$CurrentWorkMD5" == "$LastWorkMD5" ]
        then
                        if [ -e "~/.tasks.work.no" ]
                        then
                                NumOfWork=`cat ~/.tasks.work.no`
                        fi
                else
                                        NumOfWork=`w | wc -l | sed -e's/ *//'`
                                        echo $NumOfWork > ~/.tasks.work.no
                                        echo $CurrentWorkMD5 > ~/.tasks.work.md5
                fi

                if [ $NumOfWork -ne 0 ]
                then
                        LEFT='['
                        RIGHT=']'
                        Work="w:$NumOfWork"
                fi

        else
                NumOfWork=0
        fi
else
 NumOfWork=0
fi

        if [ $NumOfPersonal -ne 0 ]
        then
                if [ $NumOfWork -ne 0 ]
                then
                        SEP="|"
                fi
         fi

        RESULT="$LEFT$Personal$SEP$Work$RIGHT"

        if [ -z $RESULT ] 
        then
                RESULT="linickx@macbookpro"
        fi

        echo $RESULT
}

export PS1="\$(nick_PS1) \W \$"

... I think my next stop will be using geektool show the list/reminders on my desktop.

Lowing VirtualBox priorities

nick — Mon, 14 Feb 2011 17:30:00 +0000

One of the things I'd really like is process priorities for virtual box. In the forum I posted a couple of shell commands that I regularly type... which gets a bit tedious, following a recent article on lifehacker reviewing mac text expanding I've been prompted to automate a few things... below is a little shell script to lower the priority (renice) of all running virtual machines.

The advantage of doing this is that your host machine stays snappy, responsive and won't get too over-loaded by jobs on your VMs!

#!/bin.bash
ps -xo pid,command | grep -v grep | grep startvm | while read line ;
do
        procID=`echo $line | awk '{print $1}'`
        sudo renice +10 -p $procID
done

The above code works on a mac; although I haven't tested it, I recon to get it running on Linux you need to update the PS command, by swapping the x for an e... like this....

#!/bin.bash
ps -eo pid,command | grep -v grep | grep startvm | while read line ;
do
        procID=`echo $line | awk '{print $1}'`
        sudo renice +10 -p $procID
done

Have fun, suggestions and improvements welcome.

Bash script to fix file permissions recursively

nick — Thu, 26 Jun 2008 19:00:00 +0100

I love the redhat implementation of cron, simply drop a shell script into /etc/cron.daily/ and your script will be executed every day (by default at 4am).

Recently I've been having a small problem with mediatomb, further investigation lead me to a "Inotify thread caught exception" error which can be fixed by recursively resetting your file permissions.

What I have done to fix the issue is save the following code as /etc/cron.daily/fix_mt.sh and 'jobs a gooden' :cool:

#!/bin/bash
# Load up your mediatomb directories...
MediaTombDirectories[0]="/home/me/Videos"
MediaTombDirectories[1]="/home/me/Music"
# add more if needbe
# MediaTombDirectories[2]="/home/me/files"

# Setup find correctly.
export IFS=$'\n'

# Loop through our array.
for x in ${MediaTombDirectories[@]}
    do 
        # Find all directories & subdirectories
        for i in $(find $x -type d) 
            do 
                # Fix Permissions
                chmod -c 775 $i
                chown -c me:user $i
            done

        # Find all Files
        for i in $(find $x -type f) 
            do 
                # Fix Permissions
                chmod -c 664 $i
                chown -c me:user $i
            done
    done

Quick tip: You (or root) will get e-mailed every time this executes, you can make the script more verbos by changing all "-c" to "-v", or make it silent by removing the "-c" altogether.

Enjoy!

History Meme - What's your shell top 10?

nick — Fri, 18 Apr 2008 09:51:00 +0100

Planet gnome have started a trend...

$ history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
114 ping
107 sudo
82 ssh
71 top
71 cd
68 iwconfig
56 ls
46 usbkey.sh
43 ps
33 evolution

The output shows what commands you type the most, usbkey.sh is a script to unlock the secret-keys held on my usb key-fob, sudo is high up probably cause I use sudo to bounce services (the intel wireless drive crashes quite alot and both it and Network Manager then need restarting) and evolution needs a shutdown every now and again :( The other commands are fairly self-explanatory.

So what's your history?

Bash looping through directories & sub-directoires

nick — Tue, 05 Feb 2008 11:18:00 +0000

Note to self.... If I want to "do" something to all files in all sub-directories then I need.....

#!/bin/bash export IFS=$'\n' for i in $(find $1 -type f) do echo "$i" done