Set up a crawler login for increased ad relevancy on pages behind a login

This is the second post in our two-part AdSense Crawler Errors series.

Recently we discussed ways to improve ad targeting and relevancy by making sure that the AdSense crawler can view all pages that have AdSense code on them. Sometimes though, it can become problematic when a publisher has a members only site that requires the user to log in to view the site’s content. In cases like this, the AdSense ads that are located behind the login aren’t as relevant and targeted as they could be.

If you’re a publisher with content behind a login page, be sure to set up an AdSense crawler login so that your ads are as well-targeted as possible. Doing so will give the AdSense crawler the ability to scan content on these pages, resulting in more relevant, targeted ads that can potentially increase your CTRs and revenue. Please note that adding a crawler login will only help to deliver better, more relevant ads to pages with AdSense code already on them, and will not have any effect on which pages of your site Google Search will index.

To grant the AdSense crawler access, log in to your AdSense account and click on ‘Account settings.’ From there, scroll down to the ‘Access and authorization’ section and click on ‘edit logins’ next to crawler logins.

Then click on ‘Add a Login.’ You’ll be prompted to fill in pertinent information that the AdSense crawler will use to log in to your site and crawl your content.

After that, you’ll need to log in to Webmaster Tools to verify your website, if you haven’t already done so. Once verified, the crawler will be able to log in successfully and scan your content to deliver more targeted ads.

For additional information, take a look at our help center article regarding displaying ads on login-protected pages.

AdSense crawler errors: Check your robots.txt file for improved ad targeting and relevancy

This is the first post in our two-part AdSense Crawler Errors series.

There are many ways that publishers can go about optimizing their site for AdSense; opting-in to text/image ads, upgrading to our preferred ad formats, and increasing ad coverage across a site are just a few of the more well-known ones. But did you know that there’s another straightforward optimization tip that many publishers often overlook?

A bit of context
Your site’s robots.txt file essentially acts as a gatekeeper that determines which web crawlers, web robots, and search engines have access to your site and which do not. Those that are granted permission can do things like view your pages and index your site. Those that don’t have permission are not able to view or index specific sections of your site, depending on what you’ve specified.

AdSense ads are displayed through the use of an AdSense web crawler. That crawler scans your page’s content and determines which ads to display, according to specific keywords. If our AdSense crawler is being blocked by your robots.txt file, we’re going to have a difficult time displaying relevant ads on your site. As a result, your users may see less relevant ads, which can lead to a lower CTR.

How you can help yourself
View the contents of your robots.txt file by going to [yourdomain.com]/robots.txt. (If you have a subdomain, it likely has a robots.txt file as well, located at [sub.yourdomain.com]/robots.txt.) Be sure that the file is configured to allow our AdSense ad crawler to view your site. You can do that by simply adding the following two lines to the very top of the file:

  User-agent: Mediapartners-Google

  Disallow:

This will ensure that our AdSense ad crawler can access your site and will help display more relevant ads. As a result, you can potentially benefit from increased ad revenue. Please note that making this change will not impact your Google search rankings. Adding these two lines to your robots.txt file will only help to deliver better, more relevant ads to pages with AdSense code already on them. Pages that don’t have AdSense ad code will not be affected.

If you have URLs with any errors, you can see what they are by logging into your AdSense account and clicking on ‘Account Settings’ from the home page. From there, click on ‘View errors’ under ‘Access and Authorization.’

Stay tuned for the second post in our AdSense Crawler Errors series, where we’ll cover crawler login issues and how you can solve them.

Test your knowledge with the AdSense Academy

Whether you’re new to AdSense or have been using it for years, the AdSense Academy can help you successfully run your AdSense account by following six step-by-step learning modules. The Academy provides best practices on everything from implementing your ad units to building your traffic to controlling your ads.

The AdSense Academy includes checklists, videos and quizzes that you can take at the end of each module to test your knowledge.

	Join the Academy Now!

Please feel free to provide any feedback by commenting on this post or on the AdSense Forum

Prepare for the upcoming holiday season with placement targeting

With the holidays right around the corner, premium brand-name AdWords advertisers are preparing for the season by increasing their marketing budgets and scheduling targeted campaigns. You can prepare too by taking advantage of placement targeting to allow advertisers to directly target your site.

While all sites are eligible for placement targeting, there are things you can do to help increase the amount of placement targeted ads on your site. First, turn your custom channels into ad placements. Be sure to fill out a well-written title and description to help potential advertisers understand the audiences they’ll reach by bidding on your site. Some details to include in the description are your site name, vertical, ad position, ad size and site demographic.

Next, ensure you are using our most successful ad units and are placing them in optimal positions. Finally, claim your site in Ad Planner. Following these steps will help advertisers find your site to target and choose where to place their ads on a specific section or area. Placement targeted ads typically receive much higher RPMs, which results in higher earnings for you.

Creating targeted channels is easy and a great way to earn extra revenue, but it takes time for them to show up for advertisers, so act soon! Just follow the instructions in our Help Center. Once you've made the changes, not only will your ads be ready for this holiday season, but they'll also be targetable by advertisers for every premium campaign in the future.

Display both text and image ads for an easy change with big impact

FunCheapSF.com is an online resource for finding free or affordable events in the San Francisco Bay Area. Founder Johnny Hayes recently shared his site’s story with us and gave some tips on an easy and high impact way that he’s optimized his revenue -- by enabling both text and image/rich media ads.


Inside AdSense: How did FunCheapSF.com come about?
Johnny Hayes: I moved to San Francisco without a job and couldn’t find much work, so I had a lot of time on my hands. I spent my time hunting down cheap and free things to do in the city and, starting in 2003, began emailing my friends with a weekly list of activities. More friends found out and wanted to be added to my email list, so I decided it was time to make a real website.

IA: How did you get started with AdSense?
JH: I began using AdSense in March 2007. It was my first source of online advertising revenue. As content developed on my site and traffic grew, my AdSense revenue grew as well.

IA: Why did you decide to opt in to both text and image/rich media ads?
JH: Several months ago, I received an email from AdSense letting me know that I hadn't opted in to displaying text ads and was only displaying image ads.

I had originally chosen to only show image ads because I figured they were more aesthetically appealing to my readers. But Google’s reasoning – which made sense to me – was that even though there’s tons of advertisers using image ads, there’s a bunch more out there who are specifically bidding on text ads. If I’m only showing image ads then I’m leaving the text advertisers out of the auction. And since these text ads would only be shown on my site if they outbid image ads, why not offer this space to them?

So I decided to give the text ads a shot. It seemed like an easy thing to do that might have a positive impact on my revenue with virtually no downside.

IA: What were the results? Would you recommend this strategy to other publishers?
JH: After enabling both text and image ads on my site, I saw a 51% revenue increase and nearly a 60% increase in average eCPM. It was an easy change with a big impact. And using AdSense means I don't have to worry about the laborious task of finding and managing advertisers myself and can instead focus my time on what I like to do – writing good content, putting together a great site and growing my business. I just plugged in the AdSense code and rarely have to think about it again.

I would say to other publishers that it’s worth giving the option for both text and image ads a try. It's pretty easy to set up – I just followed the instructions in the email I had received from AdSense and it took just a few minutes. Also, set up channels to track your ads so you can see specifically which of your ad locations and ad types are monetizing well, and continue with that.

If you want to give text and image ads a try like Johnny, click here to upgrade your units now!

Once you're signed in to your AdSense account, click the "My Ads" tab in the new AdSense interface
Select all ad units
From the "Actions" dropdown, edit ad type to "Text & Image / rich media ads"
Set up custom channels to monitor the impact on your AdSense performance

Check out the images below to see where to enable both text and image ads in your account using the new AdSense interface:

Make the most of your AdSense experience with our new checklist

Starting a new program can be overwhelming. What steps should you take to get off on the right foot and make the most of your account? To help you make your way with AdSense, we're delighted to introduce the AdSense Checklist to publishers viewing our U.S. Help Center

If your language preference is set to U.S. English and you are logged in your Google Account, you'll find the checklist in the left navigation bar of the Help Center. You can recognize it by the progress bar. The checklist is divided into five parts, offering basic recommendations, required steps, and tips to help you grow your account to its full potential.

When you check an item, you'll see the progress bar update accordingly. This will help you in several ways. If you're new to AdSense, you'll get a clear overview of what steps to follow to make the most of your account. If you’re already an experienced AdSense publisher, you'll also have the chance to revisit the basics, and further develop your use of the program.

The progress bar can help you identify quickly how many of the recommended steps you’ve completed, and the checklist will suggest which step you can take next.

We hope that this Help Center feature will be useful to you and help you grow with the AdSense program! If you have other great ideas for ways to help newbies get started with AdSense or thoughts on how to make the checklist even better, join the conversation in our forum.

A deeper look at channels: URL channels (Part I of III)

As many of you know, channels are a great tool for tracking the clicks and impressions on your ad units, as well as figuring out which ad placements, sizes, and colors generate the most revenue. With that in mind, three of our very own Mountain View-based AdSense optimization specialists have created videos showcasing best practices for setting up channels within your AdSense account.

In the first episode, Matthew Carpenter-Arevalo talks about URL channels which can help you track revenue on a specific URL or domain. You'll learn how to set up a channel within your AdSense account, what metrics you can track using channels, and how to use that data to increase your earnings by generating more clicks on your ads.



You can find more information about URL channels in our Help Center. Check back next week, when we'll cover custom channels.