First, a block-all SPF txt record:

v=spf1 -all

And a DMARC policy that rejects all email that fails SPF.

v=DMARC1; p=reject; adkim=s; aspf=s;

And that’s it, quite simple steps that are enough to stop any spam emails sent from the domain.

Thanks to Alex Blackie for the excellent tip.

I took the latter route with the help of m/monit, a small open-source utility for managing and monitoring Linux systems. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.

While it may sound complicated, setting it up is actually quite straightforward.

Installation

As always, update the system first to keep the system update-to-date first.

sudo apt update & sudo apt upgrade

Then, install Monit from the Ubuntu repository.

sudo apt install monit

To verify the version after the installation,

sudo monit -V

Configure the web service

The nice thing about Monit is that it comes with a web interface for monitoring and managing the configured services. The default port is on 2812 but you can change that by modifying the configuration file.

sudo nano /etc/monit/monitrc

First, change the default monitoring interval if needed.

set daemon 120

Then, to enable and change the web interface settings, uncomment or change the following lines.

set httpd port 2812
  use address localhost    # only accept connection from localhost
  allow admin:monit        # required user credential

If you need to check the web interface remotely, comment out the localhost line and add the following instead.

allow ipaddress/32

Monitoring MySQL and Apache services

To monitor the MySQL service,

sudo nano /etc/monit/conf.d/mysql

and add the following lines:

check process mysql with pidfile /run/mysqld/mysqld.pid
    start program = "/usr/sbin/service mysql start" with timeout 60 seconds
    stop program  = "/usr/sbin/service mysql stop"
    if failed unixsocket /var/run/mysqld/mysqld.sock then restart

To monitor the Apache service,

sudo nano /etc/monit/conf.d/apache

and add the following lines:

check process apache2 with pidfile /run/apache2/apache2.pid
    start program = "/bin/systemctl start apache2.service" with timeout 15 seconds
    stop program  = "/bin/systemctl stop apache2.service"
    restart program = "/bin/systemctl restart apache2.service"

Now, reload the Monit service,

sudo service monit reload

And check the web interface now.

This cheat sheet is intended to run down the typical steps performed when conducting a web application penetration test. I will break these steps down into sub-tasks and describe the tools I recommend using at each level.

Many of the ideas presented in this sheet come from the fantastic teachings of Tim “lanmaster53” Tomes, who has kindly allowed me to share them with you here. If you or anyone you know is interested in web application penetration testing Training I highly recommend that you or your company consider Tim.

Please bear in mind that these steps are iterative so in a typical engagement you can expect to do them multiple times. This is particularly true if you manage to traverse different levels of access in an application (e.g. elevate from a regular user to an admin).

Finally, throughout this sheet, I will heavily discuss tools included in PortSwigger’s Burp Suite Professional which is a paid product intended for professional use. I apologize if this dissuades you, but at the price they offer the tool for I consider it a bargain.

/via JDow.io/

Purchasing the part and putting it on the server is easy and straightforward but getting both controllers synced up with firmware is another story.

The firmware on the server was never updated since it’s in service and the new controller has a newer version at TS251P006-02. I turned on the Partner Firmware Update and restarted the server. Then everything went sideways. The firmware sync didn’t get completed on the old controller and I got “system is unavailable” message when I tried to log into the management console. And even when I logged in through the new controller’s management port, I got this,

The only solution was to re-apply the firmware to the server. The problem was, it’s hard to find the proper version of firmware through HP website. Tried really hard for a couple of days until I read a post here.

The latest version of firmware is at TS252P005 released on Feb 9, 2017.  It’s named as “cp031212.exe”, sized at 36MB. You won’t be able to download it from HP website unless you have a warranty service contract with them. However, since HP P2000 G3 SAN is a re-branded version of DotHill AssuredSAN 3000 series (via here), you can download a copy of this firmware from DogHill, now belong to Seagate, Cloud System page, or AssuredSAN 3000 Series page. You may retain a copy of it for future use. You can also contact me if you are not able to download one from their website.

Everything was easier once I got the copy of the firmware. Install it through the management console, restart the SAN a couple times, and it’s all settled.

/update on Dec. 18, 2018/

Here is the direct download link to the firmware: TP252P005.7z

First of all, here is why RAID 5 is so bad and has stopped working since 2009.

To recap, two things to blame here.

• UREs (Unrecoverable Read Error)
• Disk storage capacity

Here is an example to illustrate why it’s so bad having RAID 5 on large capacity hard drives.

With a 7-2TB drives RAID 5 setup, when one drive failed, you will have 6 2-TB drives remaining. After you put in a new 2-TB drive, the resilver process kicks off to rebuild the array. Because the RAID controller needs to read through all remaining 6 disks, total of 12-TB of data, to reconstruct the data from the failed drive, there is a very high possibility that it will see another URE during the process. When that happens, it’s the second drive failure in the array, simply meaning Game Over.

I actually had that exact nightmare before on one of the backup servers. I ended up having to rebuild everything from scratch.

However, the theory applies to the traditional Winchester hard drives (spindle drives) that have a pretty high URE no matter how reliable it claims. What about the SSDs that slowly take over the whole world?

Surprisingly, it seems to be absolutely fine utilizing SSD drives on RAID 5 array. Here is a nice run down by Scott Alan Miller:

• SSDs generally just don’t have UREs so the second disk failure due to URE during the resilver process is non-existed.
• Time to reconstruct the data from the failed drive is hugely reduced.
• Resilver impact is much reduced as SSDs handles non-sequential data access so much better.

The only concern remaining here is the lifespan on SSD drives. For example, if you utilize 6-1TB SSDs from the same brand on a RAID 5 array, there are chances down the road when two of them died at the same time because they share the same endurance lifespan. Maybe, you could pick SSDs from a different brand with the same size or intentionally perform a disk failure to the array, e.g. sequentially hot-swap one with a spare drive in the array to differentiate their lifespans.

Now, I am marching on my way to build a new SSD RAID5 array.

A few more good readings regarding the disk array:

• http://www.smbitjournal.com/2012/12/the-history-of-array-splitting
• http://www.smbitjournal.com/2012/11/one-big-raid-10-a-new-standard-in-server-storage
• http://www.smbitjournal.com/2012/11/choosing-raid-for-hard-drives-in-2013
• http://www.smbitjournal.com/2012/11/choosing-a-raid-level-by-drive-count
• http://www.smbitjournal.com/2012/11/hardware-and-software-raid
• http://www.smbitjournal.com/2012/08/nearly-as-good-is-not-better
• http://www.smbitjournal.com/2012/07/hot-spare-or-a-hot-mess
• http://www.smbitjournal.com/2012/05/when-no-redundancy-is-more-reliable
• http://www.smbitjournal.com/2011/09/spotlight-on-smb-storage
• http://www.zdnet.com/blog/storage/why-raid-6-stops-working-in-2019/805
• http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162
• http://queue.acm.org/detail.cfm?id=1670144

The network stack does several seemingly impossible things. It does reliable transmission over our unreliable networks, usually without any detectable hiccups. It adapts smoothly to network congestion. It provides addressing to billions of active nodes. It routes packets around damaged network infrastructure, reassembling them in the correct order on the other side even if they arrived out of order. It accommodates esoteric analog hardware needs, like balancing the charge on the two ends of an Ethernet cable. This all works so well that users never hear of it, and even most programmers don’t know how it works.

Read more here.

To begin with here are a few background news articles which may or not be relevant to the happenings of last weekend…

British Airways to replace IT workers with Indian recruits flown in on temporary visas (This is Money, October 2015)

http://www.wservernews.com/go/rvpz937g/

Have you experienced IT job losses because of outsourcing? How has this affected your organization and your own job?

GMB takes concerns over British Airways IT outsourcing to MPs (Computer Weekly, January 2016)

http://www.wservernews.com/go/8vc0hki7/

Do you think that unionizing the IT profession can help prevent these kinds of disasters from happening?

Home Office ignores plight of BA techies as job offshoring looms (The Register, June 2016)

http://www.wservernews.com/go/mz5p7nzd/

Interesting that Theresa May was the Secretary for the UK Home Office at the time BA outsourced their IT operations. I wonder whether she would have been voted in as Prime Minister if this disaster had happened earlier in her Home Office watch.

BA faces IT jobs protest over offshoring (Contractor UK, Feb 2016)

http://www.wservernews.com/go/wd3mlm4p/

BA defended its decision to outsource IT operations by saying it was a “very common practice.” Is that really the case? I thought I read somewhere that outsourcing IT is on the decline not the upswing.

Now let’s look at the news as events unfolded last weekend…

British Airways cancels all flights from Gatwick and Heathrow due to IT failure (The Guardian)

http://www.wservernews.com/go/j63azckg/

British Airways faces huge compensation bill following IT crash as stranded passengers claim (The Mirror)

http://www.wservernews.com/go/hss7s6a5/

I wonder what kind of hoops customers have to jump through in the UIK in order to obtain the compensation they’re legally entitled to by law when this sort of event happens. Have any readers done this in the past?

BA’s ‘global IT system failure’ was due to ‘power surge’ (The Register)

http://www.wservernews.com/go/wa3pz7wm/

Five questions for BA over IT crash (BBC)

http://www.wservernews.com/go/akhg7wpp/

The BBC reports that BA says “The root cause was a power supply issue which our affected our IT systems.” It seems to me that the cause of the disaster couldn’t have been a power surge but inadequate systems and/or procedures to handle the possibility of such a power surge happening, right?

British Airways could face £100m compensation bill over IT meltdown (The Guardian)

http://www.wservernews.com/go/kfp35sku/

The Guardian quotes James Walker the chief executive of free flights compensation claim site Resolver as saying “This is not like an ash cloud or traffic controllers’ strike that can’t be predicted. The computer system breaking down is within its control.” Do you think that’s a fair statement given the complexity of the IT systems needed to support the operation of a large airline like BA?

British Airways flights are facing chaos for days after computer meltdown leaves more than 100k stranded (Independent.IE)

http://www.wservernews.com/go/mebifaew/

This article quotes Captain Stephen Wearing who has flown for BA for 29 years saying that last night was “the worst chaos I’ve ever seen”. Do you think our overreliance on IT systems in our modern world is setting us up for even greater chaos?

British Airways boss ‘tries to gag staff’ over IT failure which hit 300,000 passengers after ‘inexperienced (The Sun)

http://www.wservernews.com/go/afjmdjmi/

I’m not sure how reliable The Sun is for news, but I think it highly likely that IT staff are being pressured by BA’s management to keep their mouths shut over all this.

Whistle-blower claims BA travel chaos was down to dodgy computer system – but ‘bosses refused to fix it’ (The Sun)

http://www.wservernews.com/go/38intnng/

Another article from The Sun, but since the quoted source is anonymous I’m not sure if we should trust it.

BA boss ‘won’t resign’ over flight chaos (BBC)

http://www.wservernews.com/go/akvpal9u/

He won’t have to resign, he’ll get booted out for sure by shareholders pressuring BA’s board of directors.

BA flights returning to normal after damaging IT collapse (Reuters via The Daily Star)

http://www.wservernews.com/go/lxke9cee/

Did outsourcing cause the British Airways IT meltdown? (TNW)

http://www.wservernews.com/go/dxk1bamd/

The 64 million dollar question.

Commentary: British Airways has no excuse for the chaos at Heathrow airport (The Financial Times via Channel NewsAsia)

http://www.wservernews.com/go/7vsonbi3/

Anatomy of a very British Airways IT cockup (Ars Technica UK)

http://www.wservernews.com/go/o8va73mm/

Ars Technica tries to go deep but rarely gets there IMO.

What went wrong in British Airways datacenter in May 2017? (UP2V)

http://www.wservernews.com/go/n7txcq9n/

This is a much better analysis article than Ars Technica. It’s worth reading this article from start to finish. Here’s a part that grabbed our attention:

Scottish and Southern Electricity Networks (SEN), which manage the electricity distribution network in the area north of Heathrow where British Airways’ headquarters are located, said its services were running as normal on Saturday morning. “The power surge that BA are referring to could have taken place at the customer side of the meter. SEN wouldn’t have visibility of that,” a spokesman said.

Also check out this part:

From the IT rumour mill. Allegedly, the staff at the Indian data centre were told to apply some security fixes to the computers in the data centre. The BA IT systems have two, parallel systems to cope with updates. What was supposed to happen was that they apply the fixes to the computers of the secondary system, and when all is working, apply to the computers of the primary system. In this way, the programs all keep running without any interruption. What they actually did was apply the patches to _all_ the computers. Then they shutdown and restarted the entire data centre. Unfortunately, computers in these data centres are used to being up and running for lengthy periods of time. That means, when you restart them, components like memory chips and network cards fail. Compounding this, if you start all the systems at once, the power drain is immense and you may end up with not enough power going to the computers — this can also cause components to fail. It takes quite a long time to identify all the hardware that failed and replace it.

Compiled by Aaron W from Spiceworks, here are 13 free network monitoring tools that will help you detect when your network is down — and hopefully help you prevent it from going down in the first place.

1. Microsoft Network Monitor
2. Nagios
3. OpenNMS
4. Spiceworks Network Monitor
5. Spiceworks Connectivity Dashboard
6. PRTG Network Monitor
7. The Dude
8. Zabbix
9. Zenoss Core
10. Icinga
11. Pandora FMS
12. WireShark
13. Observium

There are definitely more out there, but this is an excellent list of tools Aaron compiled.