After some time in the SEO industry, basic best practices become habits, and you’ll need to dig deeper to give more value to clients. Of course, it’s always good to keep up with and refine your knowledge of best practices, but algorithm updates and research are regularly bringing us new strategies that can offer a strong competitive advantage.

Log files are nothing new, and analyzing them isn’t new either, but it’s not something most analysts do. The concept is pretty technical, and even if you can get the files and get a report generated, you might not know what to do with it.

This year when I recognized that industry leaders like Michael King (ipullrank) and Britney Muller (Moz) talking about log files, advanced technical SEO and information retrieval, I knew it was overdue to get into this. I had questions that you may have now, and this post should clear those up quickly.

What is a Web Server Log and What are Server Log Files?

A server log simply keeps track of everything happening on a server, all activities and requests included. So if a robot comes to the website and crawls a particular page, it will show that, along with other details. You can count on these files to be accurate, because they are coming directly from the network and server. Page Tagging (utilization of cookies/Javascript tracking) in-part with analytics providers isn’t as accurate.

SEO Opportunities and Improvements That Result From a Log File Analysis

• You can pinpoint where response codes are being generated, and resolve any that are concerning (4xx, 5xx)
• You can identify how frequently GoogleBot and other bots are crawling your website, along with what they’re crawling. With that, you’ll find ways to control the crawl, organize crawl budget, and make the bots work to your advantage
• You can find duplicate crawled pages
• You can notice uncrawled pages and do what is necessary to ensure they’re crawled
• You can clearly see redirect issues and resolve them
• You can identify site performance issues and improve speed
• You can explore your most and least active pages, and potentially find insights into why the bots priorities are the way they are

How to View and Analyze Log Files

The best way to view log files in a way that offers value and insight is through a tool. The standard log files are not the most readable. They include a ton of data, and you’ll need to have some technical savvy to understand them. A tool like one of the following can help you to view the files, understand what’s going on, cross-reference with other crawls, and create action items that will benefit your websites crawlability and ultimately search presence.

Helpful Tools

With most log file tools you: download log files, drag and drop, and the tool will spit out a report. Some more detailed than others. I’m a frequent user of Screaming Frog which has one, and SEMrush is also developing one, which I was excited to see. There are other tools out there, if you’ve used them and think they’re better, I’d love to hear your experience with them. Here’s the basics on SF and a preview for SEMrush:

Screaming Frog Log File Analyzer

1. Free version, but a license allows you to upload more log events and create additional projects. It’s a separate download from the main screaming frog.
2. Drag and drop raw log files
3. A report is generated (and it even organizes some key sections for you!)
4. Verify search engine bots
5. Identify crawled URL’s
6. Analyze search bot data for insights
7. Can use a variety of tools to visualize, the data, cross reference with a standard crawl, and more for specific date ranges.

SEMrush Log File Analyzer (Coming Soon)

1. Drag and drop log files
2. SEMrush will generate a report that helps identify
   1. Errors
   2. Orphan pages
   3. Most crawled pages
   4. Crawl budget and how it can be improved

Emma Labrador at SEMrush has a great log file post that is worth mention. I generally keep my posts short and sweet for those in a hurry, but if you want more detail, check out her post.

Is This Part of Your Technical SEO Strategy?

We’d love to hear about your experiences with log file analysis, and any difficulties you may have encountered.

We’re currently performing log file analysis, developing action items, and making improvements for some of our clients. In time, we’ll post updates with the results (improved crawlability, impact on rankings, etc.) If you’ve gone through this process, we’d also love to hear about the results you’ve seen too!

For the few non-SEO’s that got to the bottom of this post, webFEAT Complete can build this into your business’s SEO strategy to give rankings a boost. Contact Us if you’d like to see what we can do for your campaign.

Check out our web hosting capabilities that can help you streamline this process!

“”