Displaying httpd log file entries per minute
$ sed -n 's/.*20[0-9][0-9]:\(.*\):.. +0.00.*/\1/p' "$@" | uniq -c
Let’s say you suspect one of your web servers is misbehaving. Users are complaining that some pages are not loading. You do a
tail -f on the access log file and entries are whizzing up the screen, so it certainly seems to be doing something right.
However, your eye can’t tell the difference between 500 log entries per minute and 5000 log entries per minute.
Wouldn’t it be nice if you could see how many log file entries are being generated per minute?
With this short script – no problem!
To create a script that contains the above command and place it in your
bin directory, do the following:
$ mkdir -p ~/bin
$ cat > ~/bin/apache.byminute
sed -n 's/.*20[0-9][0-9]:\(.*\):.. +0.00.*/\1/p' "$@" | uniq -c
$ chmod +x ~/bin/apache.byminute
$ export PATH=$PATH:~/bin
You have now created a
bash shell script file called
apache.byminute and made it available as a command on your command line.
The export PATH line can be put in your
~/.bash_profile if it’s not already there, then it will be available to you next time you log in.
You can call this script directly on an apache log, e.g.
$ apache.byminute /var/log/httpd/access_log
This will give you information about all entries for every kind of invocation of the web server. I.e. it will count accesses to every php, js, css, html, jpeg, png file.
You can get more specific and useful information when you combine the command with another
grep to get only the patterns that are of interest.
$ grep 'GET /wp-admin/admin.php' /var/log/httpd/apache_access.log | apache.byminute
What’s going on here?
grep command is looking for the pattern
/wp-admin/admin.php in the file
apache_access.log. Only lines containing that pattern will be in the output of
grep. Lines recording the GETting of jpegs, css, js, etc will not be counted.
sed command is looking for the pattern
That means it’s looking for the following sequence of patterns:
1) any bunch of characters “
2) 20 followed by two digits “
3) a colon “:”
4a) any bunch of characters followed by “
4b) With the “
\(.*\)“, the backslash-escaped parentheses mark a part of the pattern for later use in the second part of the sed expression. The substitution pattern “
\1” is then used to output only the part of the string matching the pattern within the backlash-escaped parentheses
5) a colon followed by two arbitrary characters followed by a space “
6) a plus, a zero, an arbitrary character, two zeroes “
7) and another bunch of arbitrary characters “
What it spits out looks like the following:
This is the hour and minute of each log entry, separated by a colon.
This is piped through
uniq. But not just any old
uniq has the option
-c, which means “display each duplicated line only once with a count of duplicate lines before the line”
uniq -c outputs:
This is telling us that
/wp-admin/admin.php was fetched 317 times in the minute beginning at 20:57, 221 times at 20:58, only 4 times at 20:59 and 244 times at 21:00.
If you run this command over a few days’ log files you get a feel for how much traffic you normally get per minute. Any periods of unusually low (or high) traffic are easily spotted. Gaps, i.e. minutes with no traffic, can also be spotted easily. If you normally see counts in the hundreds per minute, and suddenly you’re seeing tens per minute, you know something is amiss. If some times are being output in the wrong order, you know that some entries are taking too long to process.
You can easily adjust
sed‘s regular expression to display just hours, or hours, minutes and seconds, or even hours, minutes and tens of seconds; the ideal pattern depends on the busy-ness of your server.
Because you are armed with
grep, you can issue commands to get any patterns that are of interest to you. You have massive flexibility to search for different things in a very short space of time. You can see what’s happening in real time if you use
tail -f to get the output of the log file as it appears and pipe it through the command:
$ tail -f access_log | apache.byminute
As you can see, on the Unix command line you can create extremely powerful ad-hoc tools with just a line or two of code. This kind of code is applicable to many different situations.
For analysing timestamped log entries, get used to the pattern of
- looking for interesting entries with
sed with an appropriate regular expression to cut out the pattern that is of immediate interest
uniq -c to count the occurrences of those patterns
Popcorn, Nachos und Code!
Hamburg 29. und 30. September 2016 – code.talks
Real-World Context for your Apps
The idea of small, extremely portable, digital tags, equipped with an ARM processor and numerous sensors, attachable to real world objects and discoverable by any suitably enabled smartphone app had captured my imagination. So I duly shelled out US$99 for the pre-production not yet FCC-approved Estimote(TM) Stickers Nearables Developer Kit containing ten Bluetooth Low Energy Beacons.
Developers > Post
I was excited to get my first packet of Estimote Stickers (“real world context for your apps”). This was my first encounter with beacon technology.
I downloaded the Swift tutorial and the SDK from the Estimote web site and started learning. I was finding out about two new technologies at once, the Internet of Things (IoT) in the form of the the Estimote Stickers, and Swift, Apple’s new computer language. It was a lot of fun.
On some OS X systems MAMP is unable to start the httpd.
On my system after an upgrade it was because OS X was starting its own httpd daemon.
I was able to see this by trying to start MAMP’s httpd in a Terminal:
$ cd /Applications/MAMP/bin
$ sudo ./startApache.sh
(48)Address already in use: make_sock: could not bind to address [::]:80
(48)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
A call of
lsof confirmed it:
$ sudo lsof -iTCP -sTCP:LISTEN -P -n | grep :80
httpd 576 root 5u IPv6 0x9f5f81c483ca9e1f 0t0 TCP *:80 (LISTEN)
httpd 582 _www 5u IPv6 0x9f5f81c483ca9e1f 0t0 TCP *:80 (LISTEN)
The shows two httpd processes running with process ids of 576 and 582 respectively.
To prevent OS X automatically starting its own httpd server, use the following command in a Terminal:
$ sudo apachectl stop
This has the following effects
- it stops the running httpd and
- at the next restart no httpd will be started.
After the next restart you can use MAMP to start httpd and mysqld as usual.
Developers > Post
We achieved a reduction of 59% on the download and 70% on the footprint sizes for the Unity app “Flipping Blocks” on the iOS platform.
Most of the gain came from moving graphic assets out of the build to asset bundles that are downloaded as needed at runtime.
A small additional gain was achieved by enabling Bitcode.
All numbers in MB.
|Asset Bundling the graphics
Having programmed the Flipping Blocks game and scratched the surface of Unity3d’s capabilities I started experimenting with Unity’s physics functionality. Unity does so much of the heavy lifting that I was able using about 150 lines of C# code to program a nice 3d swarming simulation based on a few simple rules:
- try to follow the leader
- don’t get too close to your neighbours
- slow down if you go too fast
- don’t get too far from the camera (leader only)
It makes your hardware work quite hard. I can watch it for ages, it’s quite fascinating.
Click here to see the Unity WebGL version.
Mike Paget at Thumthink Ltd and Peter Evans at appMotivate released the new game “Flipping Blocks” to the Apple App Store on the 3rd of December 2015.
This incredibly varied game will frustrate, captivate and delight you!
Built on the Unity3d games platform, with a Slim Framework REST API PHP backend installed on Google App Engine and Google Cloud SQL, the game runs on iPhone and iPad from iOS 8 upwards.
Download Flipping Blocks
Developers > Post
Every App needs a couple of dozen graphic assets, icons, splash screens, etc.
An excellent overview of all that’s necessary for iOS and Android is “Icons and Splash Screens”
Do your eyes glaze over when you think of all the time you’re going to spend creating all those assets? Mine do. Particularly with icons you need practically the same symbol in many different size variations for iPhone, iPad and Android.
The good news is, instead of manually creating them all in Photoshop, Gimp or whatever, you can generate them all automatically from one graphic. This service is provided by the site MakeAppIcon. Just upload one graphical template in PNG, JPEG or PSD format (preferred dimensions 1024×1024), let the site do its thing and within a short time an email with a ZIP attachment containing all the icons in every size that you wanted, and some you didn’t know you wanted.
This saves a lot of time, money and frustration and gives you more time to do something fun, profitable, or both. Many thanks to the FileSquare people for making this available free to everybody.
Developers > Post