The one line script that shows you how your web server is doing

Displaying httpd log file entries per minute

$ sed -n 's/.*20[0-9][0-9]:\(.*\):.. +0.00.*/\1/p' "$@" | uniq -c

Let’s say you suspect one of your web servers is misbehaving.  Users are complaining that some pages are not loading. You do a tail -f on the access log file and entries are whizzing up the screen, so it certainly seems to be doing something right.
However, your eye can’t tell the difference between 500 log entries per minute and 5000 log entries per minute.
Wouldn’t it be nice if you could see how many log file entries are being generated per minute?
With this short script – no problem!
To create a script that contains the above command and place it in your bin directory, do the following:

$ mkdir -p ~/bin
$ cat > ~/bin/apache.byminute
#!/bin/bash
sed -n 's/.*20[0-9][0-9]:\(.*\):.. +0.00.*/\1/p' "$@" | uniq -c
<CTRL-D>
$ chmod +x ~/bin/apache.byminute
$ export PATH=$PATH:~/bin

You have now created a bash shell script file called apache.byminute and made it available as a command on your command line.
The export PATH line can be put in your ~/.bash_profile if it’s not already there, then it will be available to you next time you log in.
You can call this script directly on an apache log, e.g.

$ apache.byminute /var/log/httpd/access_log

This will give you information about all entries for every kind of invocation of the web server. I.e. it will count accesses to every php, js, css, html, jpeg, png file.
You can get more specific and useful information when you combine the command with another grep to get only the patterns that are of interest.

$ grep 'GET /wp-admin/admin.php' /var/log/httpd/apache_access.log | apache.byminute

What’s going on here?
The grep command is looking for the pattern /wp-admin/admin.php in the file apache_access.log. Only lines containing that pattern will be in the output of grep.   Lines recording the GETting of jpegs, css, js, etc will not be counted.
The sed command is looking for the pattern

.*20[0-9][0-9]:\(.*\):.. +0.00.*

That means it’s looking for the following sequence of patterns:
1) any bunch of characters “.*
2) 20 followed by two digits “20[0-9][0-9]
3) a colon “:”
4a) any bunch of characters followed by “\(.*\)
4b) With the “\(.*\)“, the backslash-escaped parentheses mark a part of the pattern for later use in the second part of the sed expression. The substitution pattern “\1” is then used to output only the part of the string matching the pattern within the backlash-escaped parentheses
5) a colon followed by two arbitrary characters followed by a space “:..
6) a plus, a zero, an arbitrary character, two zeroes “+0.00
7) and another bunch of arbitrary characters “.*
What it spits out looks like the following:


20:57
20:57
20:57
20:57
...
20:57
20:58
20:58
20:58
...
20:58
20:59
20:59
20:59
20:59
21:00
21:00
...

This is the hour and minute of each log entry, separated by a colon.
This is piped through uniq.  But not just any old uniq.  This uniq has the option -c, which means “display each duplicated line only once with a count of duplicate lines before the line”
So uniq -c outputs:


317 20:57
221 20:58
4 20:59
244 21:00

This is telling us that /wp-admin/admin.php was fetched 317 times in the minute beginning at 20:57, 221 times at 20:58, only 4 times at 20:59 and 244 times at 21:00.
If you run this command over a few days’ log files you get a feel for how much traffic you normally get per minute.  Any periods of unusually low (or high) traffic are easily spotted.  Gaps, i.e. minutes with no traffic, can also be spotted easily.  If you normally see counts in the hundreds per minute, and suddenly you’re seeing tens per minute, you know something is amiss. If some times are being output in the wrong order, you know that some entries are taking too long to process.
You can easily adjust sed‘s regular expression to display just hours, or hours, minutes and seconds, or even hours, minutes and tens of seconds; the ideal pattern depends on the busy-ness of your server.
Because you are armed with grep, you can issue commands to get any patterns that are of interest to you.  You have massive flexibility to search for different things in a very short space of time.  You can see what’s happening in real time if you use tail -f  to get the output of the log file as it appears and pipe it through the command:

$ tail -f access_log | apache.byminute

As you can see, on the Unix command line you can create extremely powerful ad-hoc tools with just a line or two of code.  This kind of code is applicable to many different situations.
For analysing timestamped log entries, get used to the pattern of
- looking for interesting entries with grep
- using sed with an appropriate regular expression to cut out the pattern that is of immediate interest
- using uniq -c to count the occurrences of those patterns

Popcorn, Nachos und Code!
Hamburg 29. und 30. September 2016 – code.talks
code.talks

Post

Posted by Thursday August 18th, 2016 No Comments

Working with Estimote Stickers Nearables Developer Kit

Real-World Context for your Apps

The idea of small, extremely portable, digital tags, equipped with an ARM processor and numerous sensors, attachable to real world objects and discoverable by any suitably enabled smartphone app had captured my imagination. So I duly shelled out US$99 for the pre-production not yet FCC-approved Estimote(TM) Stickers Nearables Developer Kit containing ten Bluetooth Low Energy Beacons.
I was excited to get my first packet of Estimote Stickers (“real world context for your apps”).  This was my first encounter with beacon technology.
I downloaded the Swift tutorial and the SDK from the Estimote web site and started learning. I was finding out about two new technologies at once, the Internet of Things (IoT) in the form of the the Estimote Stickers, and Swift, Apple’s new computer language. It was a lot of fun.

(more…)

Developers > Post

Posted by Thursday July 14th, 2016 No Comments

MAMP can’t start httpd

On some OS X systems MAMP is unable to start the httpd.

On my system after an upgrade it was because OS X was starting its own httpd daemon.

I was able to see this by trying to start MAMP’s httpd in a Terminal:


$ cd /Applications/MAMP/bin
$ sudo ./startApache.sh
(48)Address already in use: make_sock: could not bind to address [::]:80
(48)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down

A call of lsof confirmed it:

$ sudo lsof -iTCP -sTCP:LISTEN -P -n | grep :80
httpd 576 root 5u IPv6 0x9f5f81c483ca9e1f 0t0 TCP *:80 (LISTEN)
httpd 582 _www 5u IPv6 0x9f5f81c483ca9e1f 0t0 TCP *:80 (LISTEN)

The shows two httpd processes running with process ids of 576 and 582 respectively.

To prevent OS X automatically starting its own httpd server, use the following command in a  Terminal:

$ sudo apachectl stop

This has the following effects

After the next restart you can use MAMP to start httpd and mysqld as usual.

Developers > Post

Posted by Sunday May 29th, 2016 No Comments

70% reduction in iOS app footprint of “Flipping Blocks” Unity game

Abstract

We achieved a reduction of 59% on the download and 70% on the footprint sizes for the Unity app “Flipping Blocks” on the iOS platform.

Most of the gain came from moving graphic assets out of the build to asset bundles that are downloaded as needed at runtime.

A small additional gain was achieved by enabling Bitcode.

All numbers in MB.

Measure Footprint Before Footprint After Footprint saving Download Before Download After Download saving
Enable Bitcode 385.0 371.0 14.0 83.6 79.2 4.4
Asset Bundling the graphics 371.0 117.0 254.0 79.2 34.0 45.2
TOTALS 385.0 117.0 268.0 83.6 34.0 49.6

(more…)

Post

Posted by Tuesday May 24th, 2016 No Comments

76% reduction in Unity game “Flipping Blocks” size

Abstract

We achieved a massive reduction in the size of our game app Flipping Blocks, which was developed on the Unity platform.

Here’s a summary of the measures and the corresponding reduction in download size:

Measure Before After Saving
Reduce the size of the graphical assets 105MB 79MB 26MB
Build for ARMv7 only 79MB 70.5MB 8.5MB
Remove unused assets 70.5MB 67.5MB 3MB
Use language dependent asset bundles 67.5MB 25MB 42.5MB
TOTALS 105MB 25MB 80MB

(more…)

Developers

Posted by Tuesday May 3rd, 2016 No Comments

Fun with Unity3d – Swarm simulation

Having programmed the Flipping Blocks game and scratched the surface of Unity3d’s capabilities I started experimenting with Unity’s physics functionality.  Unity does so much of the heavy lifting that I was able using about 150 lines of C# code to program a nice 3d swarming simulation based on a few simple rules:

It makes your hardware work quite hard.  I can watch it for ages, it’s quite fascinating.

Click here to see the Unity WebGL version.

Post

Posted by Tuesday January 26th, 2016 No Comments

New game Flipping Blocks developed by appMotivate

Mike Paget at Thumthink Ltd and Peter Evans at appMotivate released the new game “Flipping Blocks” to the Apple App Store on the 3rd of December 2015.

flipping-blocks-appstore

This incredibly varied game will frustrate, captivate and delight you!

Built on the Unity3d games platform, with a Slim Framework REST API PHP backend installed on Google App Engine and Google Cloud SQL, the game runs on iPhone and iPad from iOS 8 upwards.

Download Flipping Blocks

Developers > Post

Posted by Friday December 4th, 2015 No Comments

Generate all icons for iPhone, iPad and Android

Every App needs a couple of dozen graphic assets, icons, splash screens, etc.

An excellent overview of all that’s necessary for iOS and Android is “Icons and Splash Screens

Do your eyes glaze over when you think of all the time you’re going to spend creating all those assets? Mine do. Particularly with icons you need practically the same symbol in many different size variations for iPhone, iPad and Android.

The good news is, instead of manually creating them all in Photoshop, Gimp or whatever, you can generate them all automatically from one graphic. This service is provided by the site MakeAppIcon. Just upload one graphical template in PNG, JPEG or PSD format (preferred dimensions 1024×1024), let the site do its thing and within a short time an email with a ZIP attachment containing all the icons in every size that you wanted, and some you didn’t know you wanted.

This saves a lot of time, money and frustration and gives you more time to do something fun, profitable, or both. Many thanks to the FileSquare people for making this available free to everybody.

Developers > Post

Posted by Monday January 6th, 2014 No Comments

DC13 #dchh Developer Conference retrospective

Sorry, this entry is only available in Deutsch.

Developers > Post

Posted by Monday November 11th, 2013 No Comments

DC13 #dchh Developer Conference – Countdown to the record

Sorry, this entry is only available in Deutsch.

Post

Posted by Wednesday November 6th, 2013 No Comments

appMotivate offers:

  • Consulting
  • Specification
  • Project Management
  • Programming

Last blog article