Ivor O’Connor

March 9, 2013

The Universal IDE: VI with SPF13

Filed under: bash, git, howto, html5, IDE, JavaScript, Linux, mint 14 xfce, php, sqlite, ubuntu, vi, vim, vundle — ioconnor @ 4:08 am

This is a youtube video I made on how to install SPF13 on Linux Mint 14 XFCE. I like learning from youtube videos whenever possible. If in the future I want to install SPF13 and something is not how I remember it I can return to this video. SPF13 makes VI into a universal IDE. It works on most if not all OSs including windows and the mac.  Sure if you are working with something like AIX, HP-UX, or some platform that does not have vim installed SPF13 will not be for you. I’d recommend you put in your base .vimrc commands directly into the .vimrc.local file and use them along with SPF13. Picking and choosing what you find works.

UPDATE 2013-03-08: Insert the following line into your .bashrc file before starting to get full colors: export TERM=”xterm-256color”

February 18, 2013

Tutorial On Automatically Moving Home To Ram Drive And Back On Startup And Shutdown

Filed under: bash, howto, laptop, Linux, mint 14 xfce, rsync, SSD, tutorial, ubuntu — ioconnor @ 6:36 pm

This posting is meant to compliment the “Linux Mint 14 SSD Settings” and “The Best Linux Directory Structure?“posting. The idea is to move the home directory automatically at start up and shutdown to and from the ram disk. If you are using a ram disk then your computer is very fast. Possibly more importantly you are not wearing out your hard disk or SSD because all the applications that would normally write willy-nilly to the home directory are now fooled into RAM. If you don’t have the computer automatically doing these steps you might forget. So have the computer do them for you!

Firstly everybody should know how to recover in case during boot up the home directory is pointing to a non existent directory. You’ll need to boot up from a USB stick and follow these steps which I do so often I’m pulling them from memory:

  1. sudo mkdir /blah
  2. sudo mount /dev/sda1 /blah
  3. cd /blah/home
  4. sudo rm you
  5. sudo ln -s base you
  6. sudo shutdown -h now

Secondly make a log file in the home directory since this is all about mucking with the home directory

  1. sudo touch /home/log
  2. sudo chmod 777 /home/log

Thirdly follow the directions for /etc/fstab found in my “Linux Mint 14 SSD Settings” posting. I would follow all of the directions there, even for /etc/rc.local, whether or not you have a SSD drive.

Fourthly add the following code which will copy the ram drive back to the hard disk when the computer shutsdown.

Make a file called /etc/init.d/diskhome.sh and put in it something like the following but change the username:

#!/bin/sh

ivorPrintAndLog() {
tStamp=$(date +%Y-%m-%d@%T)
echo “$tStamp $1”
echo “$tStamp $1” >> /home/log
}

argUser=”ivor”
fUserOnDisk=$(ls -l /home | grep “$argUser -> base”)
#echo “fUserOnDisk: ‘$fUserOnDisk'”
fUserOnRam=$(ls -l /home | grep “$argUser -> /tmp/”)
#echo “fUserOnRam: ‘$fUserOnRam'”

if [ “$fUserOnDisk” ]; then
if [ “$fUserOnRam” ]; then # On disk and ram!
sOut=”Script needs fixing because it reports ‘$argUser’ has home on both ram and disk.”
else # On disk but not ram
sOut=”User ‘$argUser’ is already residing on disk so nothing to do.”
fi
else
if [ “$fUserOnRam” ]; then # Not disk but ram
sOut=”Moving ram contents for user: ‘$argUser’ to disk.”
ivorPrintAndLog “$sOut”
rsync -av –delete /tmp/home/base /home
cd /home
sudo rm $argUser
sudo ln -s base $argUser
cd –
sOut=”Moved information to disk successfully”
else # Not disk and not ram
sOut=”Error with script: User: ‘$argUser’ is not on disk or ram.”
fi
fi

ivorPrintAndLog “$sOut”

Make it executable and put links to it from the appropriate places

chmod +x /etc/init.d
cd /etc/rc0.d
sudo ln -s ../init.d/diskhome.sh K99diskhome.sh
cd ../rc6.d
sudo ln -s ../init.d/diskhome.sh K99diskhome.sh

Fifthly update /etc/rc.local so it copies the home directory to ram by adding the following lines into the /etc/rc.local

ivorPrintAndLog() {
tStamp=$(date +%Y-%m-%d@%T)
echo “$tStamp $1”
echo “$tStamp $1” >> /home/log
}

argUser=”ivor”
fUserOnDisk=$(ls -l /home | grep “$argUser -> base”)
echo “fUserOnDisk: ‘$fUserOnDisk'”
#fUserOnRam=$(ls -l /home | grep “$argUser -> /tmp/”)
echo “fUserOnRam: ‘$fUserOnRam'”

blah() {
if [ “$fUserOnDisk” ]; then
if [ “$fUserOnRam” ]; then # On disk and ram!
sOut=”Script needs fixing because it reports ‘$argUser’ has home on both ram and disk.”
else # On disk but not ram
sOut=”Moving disk contents for user: ‘$argUser’ to ram.”
ivorPrintAndLog “$sOut”
rsync -av /home/base /tmp/home
cd /home
sudo rm $argUser
sudo ln -s /tmp/home/base $argUser
cd –
sOut=”Moved information to ram successfully”
fi
else
if [ “$fUserOnRam” ]; then # Not disk but ram
sOut=”User ‘$argUser’ is already residing in ram so nothing to do.”
else # Not disk and not ram
sOut=”Error with script: User: ‘$argUser’ is not on disk or ram.”
fi
fi
ivorPrintAndLog “$sOut”
}

blah

Finally, the sixth step, rename your home directory to “base” and put a link to it. In my case:

    1. sudo mv ivor base
    2. sudo ln -s base ivor

That is all there is to it. The next time you reboot you should find your home directory on the tmp ram drive. Now there are a few problems. Like your home directory is too large. Or you would like to manually backup. The following commands may help:

    1. cd; du -h | sort -h
      Figure out how much space your home directory is using
    2. cd; sudo mkdir /home/Downloads; chmod 777 /home/Downloads; mv Downloads/* /home/downloads/; rm Downloads; ln -s /home/Downloads Downloads
      Basically just moving your Downloads directory which is probably rarely used to another place so it is not copied on to the ram disk. Maybe there are other directories that could also be moved because they take up lots of space and rarely get used. Like pictures, music, etc..
    3. Periodically check your ram disk usage. Mine is usually under 1GB which is less than 50% of it’s capability. (The RAM drives default to half your installed ram and the assumption is you have 4 GBs or more of RAM on your system.)
      df -h
    4. Verify everything is working:
      ls -l /home # Your home drive should be a link pointing to the ram drive.
    5. cat /home/log
      #Your log file should show the files were copied on startup and shutdown.
    6. Backup your ram drive from time to time using the bold red line above. Maybe even make it into an alias like this:
      echo ‘alias bkup=”rsync -av –delete /tmp/home/base /home”‘ >> ~/.bashrc

I’ve used these steps on a few computers and they work for me. However if you run into problems let me know and I’ll update this with the fixes.

January 4, 2013

Linux Mint 14 SSD Settings.

Filed under: howto, laptop, Linux, mint 14 xfce, SSD, ubuntu — ioconnor @ 8:41 pm

2013-01-05 UPDATE: I kept having strange random problems when I turned journaling off so I removed the parts of the script below having to do with journaling.
2013-01-06 UPDATE: Since the RAM drive is being used as the swap drive I reformatted the SSD to recover the swap partition’s space that is setup on a default installation.
2013-01-06 UPDATE: I’m using iotop, “vmstat -p /dev/sda1 60”, “sudo find / -xdev -type f -mmin -9”, and audit to identify disk activity. They go into a list. The list is then used by /etc/rc.local at startup for backup and then to RAM where they are accessed via links. At shutdown etc/init.d moves them back. I’ll keep adding to this list diligently for a few weeks until it is pretty much complete.
2013-01-08 UPDATE: Added samba and sudo directories to the fstab. Still the reads and writes according to vmstat are gradually increasing though a search with find shows nothing has been changed.
2013-01-20 UPDATE: I’m still running with these settings but use scripts to move my home directory to a ram disk and back on startup and shutdown.  Days go by without seeing the hard disk light unless I’m actively making a backup. This approach does force me to keep myself organized. I keep ISO, MP3, MP4, and other big rarely used files on external USB drives. I also keep git repositories in their own /home/git directories. Overall it seems every piece of software wants to write to my home directory willy-nilly-like. So by having my ~ on a ram drive I’m allowing my SSD drive to stay in idle mode for hours at a time.
2013-02-18 UPDATE: Added the “| sudo tee” to three lines of the script in /etc/rc.local. I noticed it was not working on some newer installs until I did that.
2013-02-18 UPDATE: Added a new post called “Tutorial On Automatically Moving Home To Ram Drive And Back On Startup And Shutdown” which nicely compliments this post.

My hard disk went bad and I was forced to upgrade. Seeing as I could get a SSD drive with more than enough space at the same price as a normal drive I got the SSD. The SSD being the Kingston HyperX 3K with 120GBs of space. (It took less than 24 hours from the time I ordered until the time FedEx handed it to me though it was shipped normal.) The 3K in the title means it can only be used for 3,000 writes to any one particular place. Three thousand writes is not a lot. It seems like I go through that many saves on each program I write.

Now they clearly say the MTBF is 1,000,000 hours. That is a tad  over 100 years. (I suppose it’s a little more believable than religion.) The warranty for parts and labor is three years. I’m almost positive they have a counter inside that shows how many times each area has been cycled allowing them to wiggle out of warranties.

So to avoid cycle times I set my system as follows. I also wrote some scripts to verify the settings took. The settings turn on trim, turn off setting access times on files, and avoids writing out to disk unless all memory is full or the computer is being powered down. I want this system to act like a big RAM drive never bothering to write changes to disk. I suspect with the following settings my SSD should last as long as any other component in the laptop. (I do need to do further research to verify my disk is not being written to by anything. Then I’ll feel truly secure.)

/etc/fstab:

#UUID=4f24887d-2948-4e13-bc4e-1e28daa4f778 / ext4 errors=remount-ro 0 1
UUID=4f24887d-2948-4e13-bc4e-1e28daa4f778 / ext4 noatime,discard,errors=remount-ro 0 1

# swap was on /dev/sda5 during installation
UUID=c85a0e3b-ae36-4f5d-9c99-f93d9c19d770 none swap sw 0 0

tmpfs /tmp tmpfs nodev,nosuid,mode=1777 0 0
tmpfs /var/lock tmpfs nodev,nosuid,mode=1777 0 0
tmpfs /var/log tmpfs nodev,nosuid,mode=1777 0 0
tmpfs /var/spool tmpfs nodev,nosuid,mode=1777 0 0
tmpfs /var/run tmpfs nodev,nosuid,mode=1777 0 0
tmpfs /var/cache/samba tmpfs nodev,nosuid,mode=1777 0 0
tmpfs /var/lib/sudo tmpfs nodev,nosuid,mode=0700 0 0

/etc/rc.local:

echo 1 > /proc/sys/vm/swappiness
echo 1 > /proc/sys/vm/vfs_cache_pressure
echo 99 | sudo tee /proc/sys/vm/dirty_ratio
echo 80 | sudo tee /proc/sys/vm/dirty_background_ratio
echo 360000 > /proc/sys/vm/dirty_expire_centisecs
echo 360000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs

December 2, 2012

SSH Over Non-Standard Ports

Filed under: cli, howto, Linux, mint 14 xfce, SSH, tutorial, ubuntu — ioconnor @ 12:35 am

My ~/.ssh/config file had something like:

Host bb
Hostname blahblah.com
Port 1234
User ioconnor

so I only had to type “ssh bb” but it’s confusing. Now I’ve changed the config file to:

Host blahblah.com
Port 1234

and use the familiar form of “ssh ioconnor@blahblah.com”. Now all additional utilities that may ride on top of ssh are not confused and can work as they were designed. Much more elegant though not as fancy.

July 25, 2009

Automatic Backup With Rsync And Cron On Ubuntu

Filed under: BACKUP, CRON, mint 14 xfce, rsync, SSH, ubuntu — Tags: , , , , — ioconnor @ 6:28 pm

Keeping a directory automatically backed up somewhere is always useful. Every few months this is needed on one machine or another. Here are the steps:

  1. First set up ssh keys so passwords are no longer needed.
    1. Test things by verifying a ‘ssh user@yourserver.com’ does require a password.
    2. Make some ssh keys on your client and then move the public key to the server
      1. cd ~/.ssh
      2. ls
      3. ssh-keygen -t dsa (Keep the defaults by just pressing enter a few times.)
      4. ssh-copy-id -i ~/.ssh/id_dsa.pub user@yourserver.com
    3. Test things by verifying a ‘ssh user@yourserver.com’ no longer requires a password.
  2. Determine the directory to be backed up and where it will be backed up and test the command to be used.
    1. time rsync -atvz ~/local-directory-of-your-choosing/ user@youserver.com:remote-directory-of-your-choosing/
    2. log in the server and verify the files are there…
  3. Put the command in cron. Because cron is broken in ubuntu you can’t simply do “crontab -e” anylonger. (How can something so basic be broken?!) Instead follow these instructions:
    1. vi ~/some_file and put in the cron commands
    2. @hourly rsync -atvz ~/local-directory-of-your-choosing/ user@youserver.com:remote-directory-of-your-choosing/
    3. crontab ~/some_file
    4. crontab -l

Now simply test it out…

UPDATED ON 2013-02-02: Used “ssh-copy-id -i ~/.ssh/id_dsa.pub user@yourserver.com” to reduce 10 or so steps and make things vastly simpler.
UPDATED ON 2013-02-05: I came across this article today and realized I should probably update this to use the “–delete” and “-e ssh” commands. I am already a big user of the delete command but I don’t often use ssh because my backups are local for the most part.

Blog at WordPress.com.