Removed ext_addr from “base_features” and commented out the inode_size line. Here is a diff from the default ubuntu e2fsprogs /etc/mke2fs.conf with the necessary changes,

[fs_types]


Then format the partition and this will allow it to be mounted on windows.

I used DiskInternals Linux Reader on windows to mount the drive and copy the files which can be downloaded for free at

I found it at


height: 120px;
overflow: auto;
font-family: “Consolas”,monospace;
font-size: 9pt;
text-align:left;
background-color: #FCF7EC;
overflow-x: auto; /* Use horizontal scroller if needed; for Firefox 2, not
white-space: pre-wrap; /* css-3 */
white-space: -moz-pre-wrap !important; /* Mozilla, since 1999 */
word-wrap: break-word; /* Internet Explorer 5.5+ */
margin: 0px 0px 0px 0px;
padding:5px 5px 3px 5px;
white-space : normal; /* crucial for IE 6, maybe 7? */


Hopefully others find this useful.

I decided that I would learn python, surprisingly it’s been really easy to pick up and I thought I would share some of the things I made in my first 2 days of python.

The first I written was to phrase the results from yahoo’s football score page(http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html). I was on this page anyway and thought why not!

#!/usr/bin/python

import urllib2
import re
response = urllib2.urlopen('http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html')
html = response.read()
re_channel = re.compile("<td class=\"ko\"><abbr class=\"dtstart\" title=\"\">([0-9a-zA-z ,:]*)</abbr></td>" +
 "\n <td class=\"match.*?\n" +
 " <a href=.*?\">" +
  "\n <span class=\"home\">([a-zA-Z ]*)</span>.*?" +
  "\n <span class=\"score\">([0-9 :-]*)</span>.*?" +
  "\n <span class=\"away\">([a-zA-Z ]*)</span>", re.I | re.S | re.M)
find_result = re_channel.findall(html)
print find_result

Sample output

[('13 Aug, 15:00', 'Wigan Athletic', '1 - 1', 'Norwich City'), ('13 Aug, 15:00', 'Fulham', '0 - 0', 'Aston Villa'), ('13 Aug, 15:00', 'Blackburn Rovers', '1 - 2', 'Wolverhampton Wanderers'), ('13 Aug, 15:00', 'Liverpool', '1 - 1', 'Sunderland'), ('13 Aug, 15:00', 'Queens Park Rangers', '0 - 4', 'Bolton Wanderers'), ('13 Aug, 17:30', 'Newcastle United', '0 - 0', 'Arsenal'), ('14 Aug, 13:30', 'Stoke City', '0 - 0', 'Chelsea'), ('14 Aug, 16:00', 'West Bromwich Albion', '1 - 2', 'Manchester United'), ('15 Aug, 20:00', 'Manchester City', '4 - 0', 'Swansea City'), ('20 Aug, 12:00', 'Sunderland', '0 - 1', 'Newcastle United'), ('20 Aug, 12:45', 'Arsenal', '0 - 2', 'Liverpool'), ('20 Aug, 15:00', 'Swansea City', '0 - 0', 'Wigan Athletic'), ('20 Aug, 15:00', 'Aston Villa', '3 - 1', 'Blackburn Rovers'), ('20 Aug, 15:00', 'Everton', '0 - 1', 'Queens Park Rangers'), ('20 Aug, 17:30', 'Chelsea', '2 - 1', 'West Bromwich Albion'), ('21 Aug, 13:30', 'Norwich City', '1 - 1', 'Stoke City'), ('21 Aug, 14:05', 'Wolverhampton Wanderers', '2 - 0', 'Fulham'), ('21 Aug, 16:00', 'Bolton Wanderers', '2 - 3', 'Manchester City'), ('22 Aug, 20:00', 'Manchester United', '3 - 0', 'Tottenham Hotspur'), ('27 Aug, 12:05', 'Aston Villa', '0 - 0', 'Wolverhampton Wanderers'), ('27 Aug, 12:30', 'Wigan Athletic', '2 - 0', 'Queens Park Rangers'), ('27 Aug, 15:00', 'Swansea City', '0 - 0', 'Sunderland'), ('27 Aug, 15:00', 'Chelsea', '3 - 1', 'Norwich City'), ('27 Aug, 15:00', 'Blackburn Rovers', '0 - 1', 'Everton'), ('27 Aug, 17:30', 'Liverpool', '3 - 1', 'Bolton Wanderers'), ('28 Aug, 13:00', 'Newcastle United', '2 - 1', 'Fulham'), ('28 Aug, 13:30', 'Tottenham Hotspur', '1 - 5', 'Manchester City'), ('28 Aug, 15:00', 'West Bromwich Albion', '0 - 1', 'Stoke City'), ('28 Aug, 16:00', 'Manchester United', '8 - 2', 'Arsenal')]

The next was to obtain a list of the uk charts top 100 and find the youtube video for it.

#!/usr/bin/python
import urllib
import urllib2
import re
import sys
from BeautifulSoup import BeautifulStoneSoup  

def getvideolink(link):
  request = urllib2.Request(link)
  request.add_header('User-Agent', "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Safari/533.3")
  response = urllib2.urlopen(request)
  text = response.read()
  videos = re.findall("<a href=\"/watch\?v=([\w-]+)", text)
  return "http://www.youtube.com/watch?v=%s" % videos[0]

response = urllib2.urlopen('http://localhost/test')
html = response.read()
re_channel = re.compile("<h3>(.*?)</h3>" +
 "\r\n.*?<h4>(.*?)</h4>.*?<h5", re.I | re.S | re.M)
find_result = re_channel.findall(html)
#remove first entry as it's a dup
del find_result[0]
for result in find_result:
 title=unicode(BeautifulStoneSoup(result[0],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
 artist=unicode(BeautifulStoneSoup(result[1],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
 link="http://www.youtube.com/results?search_query=" + urllib.quote_plus(title);
 print getvideolink(link)

Sample output

RIHANNA FT CALVIN HARRIS - WE FOUND LOVE // http://www.youtube.com/watch?v=tg00YEETFzg
FLO RIDA - GOOD FEELING // http://www.youtube.com/watch?v=3OnnDqH6Wj8
ONE DIRECTION - GOTTA BE YOU // http://www.youtube.com/watch?v=nvfejaHz-o0
LABRINTH FT TINIE TEMPAH - EARTHQUAKE // http://www.youtube.com/watch?v=u0fk6syQ7iY
ED SHEERAN - LEGO HOUSE // http://www.youtube.com/watch?v=c4BLVznuWnU
DAVID GUETTA FT USHER - WITHOUT YOU // http://www.youtube.com/watch?v=jUe8uoKdHao
MAROON 5 FT CHRISTINA AGUILERA - MOVES LIKE JAGGER // http://www.youtube.com/watch?v=iEPTlhBmwRg
JLS - TAKE A CHANCE ON ME // http://www.youtube.com/watch?v=djV11Xbc914
PROFESSOR GREEN FT EMELI SANDE - READ ALL ABOUT IT // http://www.youtube.com/watch?v=-_oLfC5Z_Ys
CHRISTINA PERRI - JAR OF HEARTS // http://www.youtube.com/watch?v=8v_4O44sfjM
LMFAO - SEXY AND I KNOW IT // http://www.youtube.com/watch?v=wyx6JDQCslE
DRAKE FT RIHANNA - TAKE CARE // http://www.youtube.com/watch?v=PaXslpx3MWY
COLDPLAY - PARADISE // http://www.youtube.com/watch?v=1G4isv_Fylg
CHARLENE SORAIA - WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
SATURDAYS - MY HEART TAKES OVER // http://www.youtube.com/watch?v=DgmoYgpMNX8
CHER LLOYD FT MIKE POSNER - WITH UR LOVE // http://www.youtube.com/watch?v=axpO86pGHAM
BRUNO MARS - IT WILL RAIN // http://www.youtube.com/watch?v=W-w3WfgpcGg
LADY GAGA - MARRY THE NIGHT // http://www.youtube.com/watch?v=O4IgYxHEAuk
LANA DEL REY - VIDEO GAMES // http://www.youtube.com/watch?v=HO1OV5B_JDw
PIXIE LOTT FT PUSHA T - WHAT DO YOU TAKE ME FOR // http://www.youtube.com/watch?v=OQCcwNMp830
KELLY CLARKSON - MR KNOW IT ALL // http://www.youtube.com/watch?v=0C_oNMH0GTk
LOICK ESSIEN - ME WITHOUT YOU // http://www.youtube.com/watch?v=lKDmJwoZ4RA
FLORENCE & THE MACHINE - SHAKE IT OUT // http://www.youtube.com/watch?v=WbN0nX61rIs
COLLECTIVE - TEARDROP // http://www.youtube.com/watch?v=u7K72X4eo_s
ONE DIRECTION - WHAT MAKES YOU BEAUTIFUL // http://www.youtube.com/watch?v=QJO3ROT-A4E
ED SHEERAN - THE A TEAM // http://www.youtube.com/watch?v=UAWcs5H-qgQ
WANTED - LIGHTNING // http://www.youtube.com/watch?v=MQyHyfLp5NI
JESSIE J - WHO YOU ARE // http://www.youtube.com/watch?v=j2WWrupMBAE
LUCENZO & QWOTE - DANZA KUDURO // http://www.youtube.com/watch?v=rUFgacK8sZ0
GYM CLASS HEROES/ADAM LEVINE - STEREO HEARTS // http://www.youtube.com/watch?v=T3E9Wjbq44E
MAVERICK SABRE - I NEED // http://www.youtube.com/watch?v=VA770wpLX-Q
WESTLIFE - LIGHTHOUSE // http://www.youtube.com/watch?v=Tivph7mTku4
SEAN PAUL FT ALEXIS JORDAN - GOT 2 LUV U // http://www.youtube.com/watch?v=tDq3fNew1rU
LADY GAGA - THE EDGE OF GLORY // http://www.youtube.com/watch?v=QeWBS0JBNzQ
GLEE CAST - RUMOUR HAS IT/SOMEONE LIKE YOU // http://www.youtube.com/watch?v=qb7zjKkLCoQ
SLOW MOVING MILLIE - PLEASE PLEASE PLEASE LET ME GET WHAT I // http://www.youtube.com/watch?v=DMQbzLrvwlE
KATY PERRY - THE ONE THAT GOT AWAY // http://www.youtube.com/watch?v=Ahha3Cqe_fk
ELBOW - ONE DAY LIKE THIS // http://www.youtube.com/watch?v=0NFV8dHrZYM
CHRISTINA PERRI - A THOUSAND YEARS // http://www.youtube.com/watch?v=z5Q8x1wYN4w
SNOW PATROL - THIS ISN'T EVERYTHING YOU ARE // http://www.youtube.com/watch?v=Q-Gljs8Y3Q8
RIZZLE KICKS - WHEN I WAS A YOUNGSTER // http://www.youtube.com/watch?v=Rc2iUwMpb8Y
GOO GOO DOLLS - IRIS // http://www.youtube.com/watch?v=NdYWuo9OFAw
DAPPY - NO REGRETS // http://www.youtube.com/watch?v=WoImizvsj5w
BRUNO MARS - MARRY YOU // http://www.youtube.com/watch?v=xB40cQD677s
ADELE - SET FIRE TO THE RAIN // http://www.youtube.com/watch?v=ss0HAdW1DnY
NICKELBACK - WHEN WE STAND TOGETHER // http://www.youtube.com/watch?v=76RbWuFll0Y
AFROJACK & STEVE AOKI - NO BEEF // http://www.youtube.com/watch?v=ksocjhxX_DQ
ADELE - SOMEONE LIKE YOU // http://www.youtube.com/watch?v=hLQl3WQQoQ0
PIXIE LOTT - ALL ABOUT TONIGHT // http://www.youtube.com/watch?v=swcULf1ATyU
NICKI MINAJ - SUPER BASS // http://www.youtube.com/watch?v=4JipHEz53sU
CALLING - WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
JAMES MORRISON - I WON'T LET YOU GO // http://www.youtube.com/watch?v=sgRb_lfIZ6A
OLLY MURS FT RIZZLE KICKS - HEART SKIPS A BEAT // http://www.youtube.com/watch?v=j5dFe-WKuPs
EXAMPLE - MIDNIGHT RUN // http://www.youtube.com/watch?v=iwYGi7YG4Js
TINCHY STRYDER/CALVIN HARRIS - OFF THE RECORD // http://www.youtube.com/watch?v=UknZiaIC9y8
SAK NOEL - LOCA PEOPLE // http://www.youtube.com/watch?v=-d6b1yn-YhQ
BRUNO MARS - RUNAWAY BABY // http://www.youtube.com/watch?v=UDG_CrqJV-0
JASON DERULO - FIGHT FOR YOU // http://www.youtube.com/watch?v=2aSOQRih6WY
CHER LLOYD - SWAGGER JAGGER // http://www.youtube.com/watch?v=sdbyG2MrBHk
LADY GAGA - BORN THIS WAY // http://www.youtube.com/watch?v=wV1FrqwZyKw
SNOW PATROL - CALLED OUT IN THE DARK // http://www.youtube.com/watch?v=GwTXwJg6_VE
WANTED - WARZONE // http://www.youtube.com/watch?v=yMR382aefmQ
JLS FT DEV - SHE MAKES ME WANNA // http://www.youtube.com/watch?v=FuwTgZOKcf8
NICOLE SCHERZINGER - TRY WITH ME // http://www.youtube.com/watch?v=R7sYiTyBjTY
COBRA STARSHIP FT SABI - YOU MAKE ME FEEL // http://www.youtube.com/watch?v=HpyZEzrDf4c
JESSIE J - WHO'S LAUGHING NOW // http://www.youtube.com/watch?v=KsxSxF3JKeU
NOEL GALLAGHER'S HIGH FLYING - AKA WHAT A LIFE // http://www.youtube.com/watch?v=lwHpLDgWonM
FOSTER THE PEOPLE - PUMPED UP KICKS // http://www.youtube.com/watch?v=SDTZ7iX4vTQ
WRETCH 32 FT JOSH KUMRA - DON'T GO // http://www.youtube.com/watch?v=bj1BMpUnzT8
BIRDY - SKINNY LOVE // http://www.youtube.com/watch?v=aNzCDt2eidg
WANTED - GLAD YOU CAME // http://www.youtube.com/watch?v=2ggzxInyzVE
BIRDY - PEOPLE HELP THE PEOPLE // http://www.youtube.com/watch?v=OmLNs6zQIHo
NICKI MINAJ FT RIHANNA - FLY // http://www.youtube.com/watch?v=3n71KUiWn1I
SKREAM FT SAM FRANK - ANTICIPATE // http://www.youtube.com/watch?v=O3Z1X4MPsqk
WILL YOUNG - JEALOUSY // http://www.youtube.com/watch?v=9MHtrM-jf9o
ADELE - ROLLING IN THE DEEP // http://www.youtube.com/watch?v=rYEDA3JcQqw
JESSIE J FT BOB - PRICE TAG // http://www.youtube.com/watch?v=qMxX-QOV9tI
LMFAO/LAUREN BENNETT/GOONROCK - PARTY ROCK ANTHEM // http://www.youtube.com/watch?v=KQ6zr6kCPj8
JASON DERULO - IT GIRL // http://www.youtube.com/watch?v=4oGUHRXT-wA
BEYONCE - COUNTDOWN // http://www.youtube.com/watch?v=ACkBTqwxcUI
RIZZLE KICKS - DOWN WITH THE TRUMPETS // http://www.youtube.com/watch?v=-aY92XgykhU
COLDPLAY - EVERY TEARDROP IS A WATERFALL // http://www.youtube.com/watch?v=fyMhvkC3A84
CALVIN HARRIS - FEEL SO CLOSE // http://www.youtube.com/watch?v=dGghkjpNCQ8
ADELE - MAKE YOU FEEL MY LOVE // http://www.youtube.com/watch?v=LLoyNxjhTzc
ONE DIRECTION - ANOTHER WORLD // http://www.youtube.com/watch?v=RyZfNBVX1q0
COLDPLAY - VIVA LA VIDA // http://www.youtube.com/watch?v=dvgZkm1xWPE
BEYONCE - LOVE ON TOP // http://www.youtube.com/watch?v=Ob7vObnFUJc
EAGLE-EYE CHERRY - SAVE TONIGHT // http://www.youtube.com/watch?v=dTa2Bzlbjv0
EN VOGUE - DONT LET GO (LOVE) // http://www.youtube.com/watch?v=QUdAT5Fwnvk
KATY PERRY - FIREWORK // http://www.youtube.com/watch?v=QGJuMBdaqIw
MODESTEP - TO THE STARS // http://www.youtube.com/watch?v=UTKSUlMbp9A
DRAKE - HEADLINES // http://www.youtube.com/watch?v=cimoNqiulUE
ED SHEERAN - YOU NEED ME I DON'T NEED YOU // http://www.youtube.com/watch?v=temYymFGSEc
WOODKID - IRON // http://www.youtube.com/watch?v=vSkb0kDacjs
DELILAH - GO // http://www.youtube.com/watch?v=cxNe9jWNuEU
KATY PERRY FT KANYE WEST - ET // http://www.youtube.com/watch?v=t5Sd5c4o9UM
JAMES VINCENT MCMORROW - HIGHER LOVE // http://www.youtube.com/watch?v=9Z-fE1l9SZ4
BRUNO MARS - JUST THE WAY YOU ARE (AMAZING) // http://www.youtube.com/watch?v=LjhCEhWiKXk
PITBULL/NE-YO/AFROJACK/NAYER - GIVE ME EVERYTHING // http://www.youtube.com/watch?v=EPo5wWmKEaI

Then I wanted to test threading so I came up with something to check a list of links and obtain the "Server:" header

#!/usr/bin/env python
import urllib2
import re
import os
import sys
import time
from multiprocessing import Pool

#Accept file name as input or use default
try:
  filename = sys.argv[1]
except IndexError:
  filename = "links"

#Function to load urls from file
def loadurls(filename):
  try:
    inputdata = file(filename).readlines()
    if len(inputdata) > 0:
      return inputdata
  except:
    print "ERROR - Unable to process url list"
    sys.exit()

#Function to get SERVER header
def getversion(url):
  try:
      request=urllib2.Request(url)
      response=urllib2.urlopen(request)
      version=response.info().getheader('Server')
  except:
      return "ERROR - Unable to fetch %s" % (url)

  if version: print "Url: %s Version: %s" % (url, version)

start = time.time()
urls=loadurls(filename)
p = Pool(15)
data = p.map(getversion,urls)

elapsed = (time.time() - start)
print "Processed %s urls in %0.2f seconds" % (len(urls), elapsed)

Sample output

scott@scott:~/python$ ./server-header.py list
Url: http://www.bbc.co.uk
 Version: Apache
Url: http://www.google.co.uk
 Version: gws
Url: http://www.scottmcintyre.net
 Version: Apache
Processed 4 urls in 1.07 seconds

None of these really serve any purpose and were merely tests I thought I would share.   Overall things have been quite easy and feel reasonably confident after just 2 days.

For those of us that don’t have the luxury of using all transactional based tables such as InnoDB and the the ability to use the likes of –single-transaction with –master-data we do have an option of using r1soft to avoid lengthy downtimes in the event we have to setup/resetup a slave. You can turn a potential several hour outage in a 30-60 second READ LOCK and perform the rest in the background.

Firstly a quick explanation of the r1soft mySQL addon, the mySQL addon works by issuing a global lock and then flushes changes from memory to disk to ensure consistent snapshots. As it works with the r1soft device driver the process usually takes less than a second making this the perfect method to take backups and have the master position at the time the backup was taken. At the time of writing r1soft does not support the ability to store the master/slave status data although hopefully in the future it will store and display it to avoid having to use the following trick.

We can however work around this by monitoring the r1soft server-log.txt to determine when it has flushed and unlocked the mySQL instance. As it happens very early on in the process we can manually issue a global read lock, get the current position, monitor the server-log.txt until r1soft has performed the snapshot of mySQL and then release the global lock manually.

I have performed an example of this for this article. I started by logging into the CDP interface, browsing to the system in question and getting to the start backup process, having this all filled in and ready to submit speeds the process up. Once this was ready I logged into the database system and issued a global read lock then obtained the current master position,

mysql> flush tables with read lock;
Query OK, 0 rows affected (0.10 sec)

mysql> show master status;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| mysql-bin.024272 | 120146094 | | |
+-----------------+-----------+--------------+------------------+
1 row in set (0.00 sec)

I then clicked ok under the schedule backup window and started a tail on server-log.txt (usually found in /usr/r1soft/buserver/log) and waited until it reached the mySQL lock, flush and unlock.


2011-01-27 02:44:27,743 INFO: (tid:976019) Starting task 'Backup' for host (xx.xx.xx.xx) with host id (xxxxxxxx)
2011-01-27 02:44:27,771 INFO: (tid:976019) Starting Backup Task.
2011-01-27 02:44:27,772 INFO: (tid:976019) Backing up partition tables.
2011-01-27 02:44:27,773 INFO: (tid:976020) Starting task 'Partition Table Backup' for host (xx.xx.xx.xx) with host id (xxxxxxxx)
2011-01-27 02:44:27,790 INFO: (tid:976020) Starting partition table backup for host (xx.xx.xx.xx) device (/dev/sda) Disk Safe (0) recovery point (6025).
2011-01-27 02:44:27,892 INFO: (tid:976020) AGENT: (Righteous Backup Linux Agent) 1.62.0 build 6386
2011-01-27 02:44:28,019 WARN: (tid:976020) AGENT: allowing control from backup server (xx.xx.xx.xx) with valid RSA key
2011-01-27 02:44:28,059 WARN: (tid:976020) AGENT: sending auth challenge for allowed host at (xx.xx.xx.xx) port (41758)
2011-01-27 02:44:28,108 INFO: (tid:976020) AGENT: host (xx.xx.xx.xx) port (41758) authentication successful
2011-01-27 02:44:28,193 INFO: Agent (xx.xx.xx.xx) authenticated successfully
2011-01-27 02:44:28,267 INFO: (tid:976020) AGENT: Partition Backup request accepted. Starting backup.
2011-01-27 02:44:28,269 INFO: (tid:976020) AGENT: Found 4 partition(s), 1 extended partition(s) and 124 extra sector(s)
2011-01-27 02:44:28,270 INFO: (tid:976020) AGENT: Need to back up 126 sectors
2011-01-27 02:44:28,270 INFO: (tid:976020) Sector size (512) bytes.
2011-01-27 02:44:28,271 INFO: (tid:976020) Number of sectors (126).
--SNIP--
2011-01-27 02:44:32,083 INFO: Sending (1) MySQL instances to agent.
2011-01-27 02:44:32,083 INFO: Sending next MySQL instance to agent.
2011-01-27 02:44:32,117 INFO: (tid:976019) AGENT: Backup request accepted. Starting backup.
2011-01-27 02:44:39,340 INFO: (tid:976019) AGENT: Flushed 1 MySQL instances.
2011-01-27 02:44:39,353 INFO: (tid:976019) AGENT: Flushed 1 MySQL instances.
2011-01-27 02:44:39,354 INFO: (tid:976019) AGENT: Flushed 1 MySQL instances.
2011-01-27 02:44:39,355 INFO: (tid:976019) AGENT: Locked 1 MySQL instances.
2011-01-27 02:44:39,355 INFO: (tid:976019) AGENT: Unlocked 1 MySQL instances.
2011-01-27 02:44:39,356 INFO: (tid:976019) AGENT: Snapshot completed in 0.059 seconds.
2011-01-27 02:44:39,357 INFO: (tid:976019) AGENT: Acquired MySQL lock in 0.025 seconds and held lock for 0.085 seconds on 1 instance.
2011-01-27 02:44:39,357 INFO: (tid:976019) Backup pipeline size (256) blocks.
2011-01-27 02:44:39,358 INFO: (tid:976019) Block size (4096).
2011-01-27 02:44:39,358 INFO: (tid:976019) Partition size (34529701) blocks.
2011-01-27 02:44:39,359 INFO: (tid:976019) Mount Point (/).
2011-01-27 02:44:39,803 INFO: (tid:976019) Network Queue Size: 1280 blocks

Once it has completed the stage in bold I simply released the read lock,

mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)

I now had a backup in r1soft and had the master position of the time it was taken. Once the backup completed I simply restored the tables I needed to the slave using the restore to an alternate host and started replication using this data.


INFO 01/27/2011 02:44:27 GMT (tid:976019) Starting task 'Backup' for host (xx.xx.xx.xx) with host id (xxxxxxxx)
INFO 01/27/2011 02:51:10 GMT (tid:976019) Backup Task finished.

I restored the backup to the mysql datadir, as you are backing up the master you can skip all the binlogs. After this was restored I started mySQL, changed the master to the details we got from the master status earlier and started the slave.


mysql> CHANGE MASTER to MASTER_HOST='192.168.1.3', MASTER_USER='slave', MASTER_PASSWORD='xxxxxx', MASTER_LOG_FILE='mysql-bin.024272', MASTER_LOG_POS=120146094;
Query OK, 0 rows affected (0.01 sec)

mysql> show slave status \G;
Seconds_Behind_Master: 0


This blog post will be about what I have learned in 2010 and some plans for 2011.

So what have I learned, enjoyed and hated in 2010?

- I love puppet

- Adobe flash is the bane of my life. Flash on 64bit linux is nothing short of a pain, it causes the most problems, the most crashes and should rot in hell.

- I am starting to like chrome but have not made the full switch from firefox yet.

- I am quite a lazy programmer, more worrying I actually don’t care.

- I love MongoDB and expect big things from it.

- I am unsure weather to love or hate r1soft. On one hand I like what it brings on the other hand it’s very buggy and I don’t think the release of 3.0 is going to address many of these. This should be an entire blog post in itself later.

- Started using ksplice uptrack, already have this running on a few hundred systems and no problems to date. Only time will tell.

I am looking forward to 2011, I am hoping to continue on with getting large portions of the “todo list” completed. Some of the ones off the top of my head are,

- Make the switch from cacti to collectd+MongoDB. Been working on this on and off for awhile. Primary goals is to make it more flexible, more automated and easier to maintain. Since we maintain so many different environments it has to fit some very specific needs. Hoping to replace the primary cacti install and also some of the other secondary customer-specific ones (such as munin).

- Upgrade to nagios 3.x. We still are running nagios 2.x so an upgrade is needed. At the same time need to rebuild and rewrite the automation systems, templates, hosts and groups. Especially to tie better into collectd, the existing nagios+cacti setup is pretty “hacky”.

- Rebuild the client database, notes and task list systems. This work is already under-way so should be completed in Q1 2011. Better encryption, more functionality, more flexibility, easier to manage.

- Build a new ticket system and drop Kayako. I don’t particularly like the direction Kayako has taken in their latest release so it’s time to stop using it (Very hacked up anyway).

- Better documentation. I, along with many others, really fail in this area and I would love to improve on it. Still toying with the idea of internal wiki’s combined with the existing notes systems and to make better use of them.

- Improve the existing yum repo, make more use of mocks, include some newer RPM’s and update all systems to use this. Replace all custom builds to an RPM from the repo (nginx, memcached, etc).

- Make more use of the nessus systems and plugin feeds. At the moment this is mostly semi-manual , need to work on making it fully automated and easy to maintain.

- Post more blog entries? Goal of atleast 20 more blog posts (Less than 2 per month).

- Rewrite the nagios r1soft/buagent check and release it.

- Make the vbulletin static cache into a plugin. This is just a simple patch we provide to customers who don’t have enough capacity and are receiving a sudden burst of traffic (IE a thread slashdotted/etc). It simply stores the threads for guests in memory for 2 minutes and then displays the cache for guests (supports online counters, multiple themes, browsers, mobile, etc).

That’s my list for the time being. Lets hope I can stick to it. Do you have your list ready? Feel free to share.

On Friday I definitely noticed something was not quite right but I was not sure, as the day progressed I felt that the left side of my face was being odd and my eyes were hurting. In a conference call I really noticed it as I started to speak and mentioned in IRC that I was having difficulty speaking clearly.

[17:25] <Scott-Mc> something is definitely wrong ,  even struggling to talk with the mic on the left side of my face:|
—– then 10 minutes later —–
[17:36] <Scott-Mc> god knows what is wrong with me, but I am even struggling to speak

After the conference was finished at about 6PM I started to think about all the things over the past few days and realized something wasn’t quite right and maybe it just wasn’t a bump to the head. So I decided to call the NHS direct 24/7 helpline and after describing my symptoms and answering some questions I got put through to a nurse. She started asking more specific questions because previously I had been unsure exactly what was wrong or how to describe it but within about 5 minutes she had determined what was wrong, told me what it was and had arranged for me to visit the doctor at the A&E.

During the call she asked me to go stand in front of the mirror and smile, it was then I realized that it wasn’t the left side of my face at all – this was perfectly normal, it was the right side. It was paralyzed and it was at this moment I freaked out, especially because she said I should go to the A&E just now.

It was very helpful just to speak to someone as otherwise I would have waited till Monday to go to the doctor if it had not cleared up but speaking to someone made me go immediately. The symptoms did rapidly get worse between about 2PM-6PM as I never noticed it when I looked in the mirror earlier.

I arrived at the A&E about 20 minutes later and waited 3-4 minutes to see a doctor. Again I described my symptoms and he told me I had Bell’s Palsy and explained it alittle. He prescribed me steroids for 1 week and some eye drops. When I asked if he knew how long it would take he said and I quote, “I wont lie, it can take awhile”. I was unsure what to make of this but when I returned home it was straight onto google to read more on it.

As it turns out the vast majority recover from it (85%) and they don’t know what causes it. So far it’s been 8-9 days and I would say that on the fourth day it showed signs of improvement that I definitely noticed (I could smile slightly more) but since then I don’t think it’s improved any which is alittle disappointing. I would really like it to go away ASAP!

At this point I have another appointment with the Doctor on Monday to check the progress, I am hoping it’s cleared up fully in about 2-3 weeks. I hope this is realistic but to be honest it doesn’t really effect me all that much. As I work from home I don’t notice it, the only thing I notice is my right eye can be alittle painful as I have to manually blink it but this has been improving in the past few days also. Overall I have a feeling I should treat this a serious thing but I have not been, still can’t believe it’s been over a week with it already (I was going to blog about it the day after).

What is surprising is the number of number of friends and friends of friends have had this, especially as I had never even heard of it before last week, the good news is that all of them it cleared up just fine, the bads news is that one of them took 6 months.

So in effort to find the initial cause of this with no real data to work from I had a search around the system for the usual suspects but couldn’t find any particular culprit. Manually inspecting nearly 2 million files was not an option so opted to carry on with the usual setup and enforced posix ACL’s against the apache user and setup some more explicit bandwidth monitoring to obtain data when an attack was occurring.

A few hours later an alert came in that the outbound bandwidth exceeded the threshold so I promptly begin investigating, The process list doesn’t seem to show any obvious usual culprits. After spending a few minutes with iftop and tcpdump I identified the targeted IP and that the traffic was being directed to a DNS server (port 53). I filtered traffic to this IP while investigating the source, as there was no unusual processes I decided to have alook at the apache status and found the GET request containing the destination IP and port (xxx.php?target=xx.xx.xx.xx&port=53).

I got the vhosts path from the httpd.conf and reviewed the file and it looks like a simple php script to perform a UDP flood to the target,

$sock=socket_create(AF_INET,SOCK_DGRAM,SOL_UDP);

while(true)
{
if(!socket_sendto($sock,$data,strlen($data),0,$target,$port)) die("Error SendTo!!!");
}


That is a snippet of it. It is very unusual to see these PHP based which is the reason for this blog entry and a definite new addition to the search list.

For those of you interested the customer already had MRTG installed and below is the last week which shows the attack saturating the uplink (100Mbit) when it was occurring but has since been stopped.

While I work with and “hack” linux all day long modifying my phone was something I have never had the real urge to do.  I very rarely use my phone as I always either have or am close to a computer/laptop and the t-mobile 3g signal is useless.  I decided as I have some free time today that I will give it a go anyway – to my surprise it is actually surprisingly easy,  all of the work is already done for you amongst the very large community of users.

I initially looked at these 2 forums,

forum.xda-developers.com
modmygphone.com

Which then I simply ended up using

http://www.ryebrye.com/blog/2009/08/16/android-rooting-in-1-click-in-progress/ but did also read http://androidandme.com/2009/08/news/how-to-root-a-t-mobile-g1-and-mytouch-3g-android-phone/

Overall I am still surprised how much of the work is actually done already.   One thing that worried me was the differences between the US and UK in the phones themselves as I do not pay much attention to phone modding this is something that could potentially be a problem, I wasn’t sure if I should be worried about it (Especially with all the disclaimers about being targeted for the US version only) but it seems to have worked fine.  I only ran into 1 issue which had nothing to do with that and I will describe the details below below.

I mostly followed the ryebrye instructions and this is what I did.

-  On my laptop I downloaded http://g1files.webs.com/Zinx/flashrec-20090815.apk ,  copied this over to the SD card.
-  Used the market to find a file browser as I couldn’t see one by default.  The one I downloaded was FileDroid Lite.
-  Installed flashrec,  opened it,  clicked “Restore Backup Recovery Image” then  “Flash Cyanogen Recovery 1.4″

I then rebooted the phone into recovery mode by turning the phone off,  then holding the home button and the power button at the same time.   The recovery system did appear as described,

I selected the “nandroid backup” at this stage to backup the phone settings.  At this point I had thought that was it completed but this is actually only to be able to load the images.  So I booted the phone again.    Then on my laptop I downloaded, update-cm-4.1.99-signed.zip from http://n0rp.chemlab.org/android/experimental/ and copied this to the sd card.  I rebooted the phone again back into recovery mode and selected “wipe data/factory reset” and then “apply any zip from sd” and selected update-cm-4.1.99-signed.zip.   After hitting home and waiting for it to install I rebooted the phone.  It was said to be patient as it takes longer to boot but after 20 minutes I realized something was up as it was still stuck at the t-mobile logo.   I powered it back into recovery mode and instead of selecting “restore latest backup” I accidentally selected nandroid backup again which I paniced, aborted (pulled the battery out) but after turning back into recovery mode it was unable to restore the good image as it had already been overwritten.   At this point I had thought that I have bricked my phone as without a card reader I have no way to put an image to the SD card but as it turns out you can mount the SD card from the recovery console.

So I selected “go to console” from the menu and plugged the data cable back in to my laptop then executed,

echo /dev/block/mmcblk0 > /sys/devices/platform/usb_mass_storage/lun0/file

Which done the trick,  I was able to copy new roms to the sdcard and get my phone back and working.  From reading the forums (This thread) I read the instructions again and noticed signed-dream_devphone_userdebug-ota-14721.zip was required from http://developer.htc.com/adp.html#s3 so I copied this to the SDcard and booted back into recovery mode.

I selected “apply any zip from sd” and then signed-dream_devphone_userdebug-ota-14721.zip , once this was installed and the phone rebooted (I held home again) I selected “apply any zip from sd” again then selected the previous image (update-cm-4.1.99-signed.zip). Once this finished installing I rebooted the phone and success! it actually booted this time. After filling in the gmail info everything seems to work fine and all my contacts are there.

The first thing I done was check the console which this rom comes with a nice app “Terminal Emulator”, I executed “su” and was prompted to allow the superuser permissions (I assume this is from the “superuser permissions” app which seems to be installed and low and behold I was root.

Overall very easy process so a huge thanks to all the people that I linked to during the course of writing this. Hopefully I will get some time to play around more and write some scripts/apps that will benefit me and others.

Not knowing what to write about is generally what made me always not bother and then when there is a subject I am actually not that good at being creative.  The truth is though it is actually just lazyness that has stopped be.

So from today onwards I have decided to have an average of 2 blog posts per month and these can be about anything be it useful information,  a rant about a product/service or just a general meaningless post.

I have made a quick guide on how to setup a server specifically for MFA sites from start to finish. The server will consist of,

• Web server – Apache
• Database – mySQL
• Scripting Language – PHP
• FTP Daemon – vsFTPD
• DNS Server – Bind

We are going to assume you are using centOS and have yum available. This should only be used on a fresh install. So lets begin,

Installing Services

To make this quick we are just going to use the RPM’s available from the depositories. At the shell prompt execute,


yum -y install httpd httpd-devel mysql mysql-server mysql-devel vsftpd bind bind-libs bind-utils php php-cli


Once this completes you should see something like this,

Now that the services are installed it’s time to configure them.

Configure Apache

Depending on which centOS version you are using you will either have apache 1.x (centos4) or apache 2.x (centos5) we are going to assume apache 2.x however as the change are only minor config changes they are the same anyway.

Rather than opening and configuring we will just do a quick replacement on the file for the NameVirtualHost and to include a single file in which we will keep all the vhosts.


perl -pi -e 's/#NameVirtualHost \*:80/NameVirtualHost \*:80\ninclude conf\/domains.conf/g' /etc/httpd/conf/httpd.conf


This will edit the config for you and if you view the config you should see something like this

Configure DNS

When configuring the DNS we are only going to setup 1 domain which will be the primary nameservers, we will use masterdomain.com as an example throughout. There will not be any domains actually setup other than the primary nameserver domain as these will be setup later with the script we build.

Setup the standard configuration (recreating as centOS 5 does not provide /etc/named.conf)

cat >/etc/named.conf <<EOM
// Default named.conf generated by AdminGeekZ

EOM


Now setup the nameservers we will be using the following for our example,

- ns1.masterdomain.com -> 10.1.1.1
- ns2.masterdomain.com -> 10.2.2.2


cat >/var/named/masterdomain.com.db <<EOM
$TTL 6200
@ 6200 IN SOA ns1.masterdomain.com. server.masterdomain.com. (
2006070312
6200
7200
1419200
6200
)

/etc/init.d/named restart


Now we create a dns zone template, this is for our bulk setup script later.


cat >/etc/template.named <<EOM
$TTL 6200
@ 6200 IN SOA ns1.cdomain.com. server.cdomain.com. (
2006070312
6200
7200
1419200
6200
)

cdomain.com. 6200 IN A 10.1.1.1
EOM


Your /etc/named.conf should look something like this now,

That’s all that is required for this section of bind now.

Setting up FTP/SSH Account

For ease of use we will have all domains managed by one FTP account of the username node which accesses /home/httpd/domains


mkdir -p /home/httpd/domains
adduser -d /home/httpd/domains node
echo "my??password12" | passwd node --stdin


You will have to chown and chmod this directory later for extra security.

Creating Setup Script

We will now use a script which can be used to add domains easily, this will use the base we already setup earlier.

The Script

cat >/root/setup.sh <<EOF
#!/bin/bash

EOF
chmod 700 /root/setup.sh


That’s the script created and to add a domain you simply run

/root/setup.sh domain.com

Bulk Adding Domains

Now that everything is setup and ready the last part is to bulk add all of your domains. To do this we are going to have them in a list (without www.) so have a file called domains.txt which should look something like this,

google.com
msn.com
yahoo.com

When you have your list and want to setup the domains first comment out the apache and named reload lines from the /root/setup.sh script to make this go much faster (you can reload once completed) and then loop through the domains by using something like this,


for i in `cat domains.txt`;do /root/setup.sh $i;done


Once completed reload named and apache


/etc/init.d/named reload
/etc/init.d/apache reload


Final Touches

Now that your server is setup and you can add domains easily the last thing to do is to start all the services and make sure they start on boot. You may wish to optimize the server aswell as any other misc tweaks (Such as adding index.php to the DirectoryIndex).


/etc/init.d/httpd restart
/etc/init.d/mysqld restart
/etc/init.d/vsftpd restart
/etc/init.d/named restart
chkconfig httpd on
chkconfig mysqld on
chkconfig vsftpd on
chkconfig named on


Summary

We setup the following,

• 3 Domains (google.com/msn.com/yahoo.com) for both dns and web
• A master FTP account (username: node / password: my??password12)
• 1 master nameserver (masterdomain.com) which all domains use

If you followed this from start to finish you should now have a fully working server for your MFA sites where you can add new domains easily and manage all the domains from one account.