Log Import Basics

Here’s how it works:

• your web server serves your visitors and writes the pages they get to the log;
• once a day or so the logs are “rotated”, so the web server starts using a new file for logging and the previous file is left behind;
• that’s when you can get that file and feed it to Log Analytics for analysis;
• then you can delete the log file or keep it somewhere safe just in case.

Log files are usually rotated on a daily basis, but you can configure that in your web server settings. It might be a good idea to setup the log rotation, so they are compressed and put into a separate folder and deleted in a month or so. This way you have a good amount of user activity data and if something goes wrong with importing data to Log Analytics you can always re–create everything from scratch.

Importing a Log File

In order to import a log file, you run Log Analytics with the import option and mention the property you want to import to:

./analytics import "Flower Shop"
analytics.exe import "Coffee Shop"

However, if you simply run it this way, you will see nothing. The software will simply hang and will be waiting for the input.

The reason is that the software expects the log data to come from the standard input, so the actual code should be like this:

cat log-file | ./analytics import "Flower Shop"

This way the log file is sent to the Log Analytics executable, where it is parsed and imported into the “Flower Shop” property as per the command line parameters provided above.

Processing Compressed Log Files

Say your web server compressed the logs using Gzip. The log files will have .gz extension then. The software doesn’t understand (yet) the compressed logs, so you need to help it this way:

gunzip -c log-file.log.gz | ./analytics import "Flower Shop"

There is many other options to pass compressed files to the standard input of applications in Unix, as well as in the modern Windows operating system.

Reverting Import

Unfortunately, there is no way to revert the import if the data gone to a different property. Double-check the log files you feed to the software and the property names you use in order to avoid mistakes.

If you accidentally imported data to the wrong property and don’t have a backup of the database, the easiest way to fix that is to delete the database, re–create the property and re–import the log files again. That’s why it’s a good idea to keep the log files around as well as make backup copies.

Server–specific Tutorials

The only supported web server at the moment is Apache, we’re going to add more in the future. Below is the list of supported servers and their specific tutorials:

• Importing Apache Logs