Your site has been switched to Mobile First Indexing

Neo · September 10, 2019, 7:34am

Well, Google throws the web a curve ball again:

I thought I was going to get a break from coding; but no.....

Neo · September 11, 2019, 12:46am

Update:

After sleeping on this change by Google (which we cannot reverse or control), and looking at Google Analysis as well as Google Search Console, I am not going to make any immediate code changes because all the stats show traffic continue to rise month-over-month (since April 2019). Our index of links with Google continues to grow, month over month.

So, I think better not to make any major changes based on the Google change to "Your site has been switched to Mobile First Indexing" and monitor the situation week by week.

I also checked to see if there is any "much better" responsive forum software (that works great on both mobile and desktop) to port our DB over to; for example the latest version of vBulletin, Xenforo, etc. But after a quick look, I did not see any compelling reason to do this; because the issue with Google seems to be with forums in general.

If there was any modern, very good, forum software which looked great (mobile and desktop), was well supported and had the tools to port our legacy vB database over to the new forum, I would consider this; but I do not think there are any good options out there.

So, I better not make any major knee-jerk reactions (code changes) to something Google has done which I don't really understand well.

If anyone has more information or suggestions, based on knowledge of what Google is doing here, please reply.

Akshay_Hegde · September 11, 2019, 2:00am

Google likes, if pages are W3C and WCAG compliant.

Few links may be useful.

Make your site more accessible - Sites Help

Web Accessibility Checker
IDI Web Accessibility Checker : Web Accessibility Checker

W3C validator
The W3C Markup Validation Service

Probably if we use PageSpeed Modules many markup can be corrected, also can reduce few http requests, if you enable http2 (major modern browsers supports) protocol then we may gain little improvements in performance ( no need to minify resources, multiplexing takes care )

HTTP/2: the difference between HTTP/1.1, benefits and how to use it

Neo · September 11, 2019, 2:12am

Hi Akshay,

All of those things you listed, we did in 2018 and before with the exception of mod_pagespeed , for Apache2, which broke the site the last time I tried to implement it.

However, I admit that we need to spend more time optimizing the mobile HTML since most of the optimization has been on the desktop.

But the issue is really "why"? Because Google Analytics shows that a full 94% of our site traffic is from desktop browsers and only 4% is from mobile browsers, so it seems a bit "nutty" for Google to automatically switch our indexing to "Mobile First Indexing" when mobile accounts for only around 4% of all traffic.

I will try to enable mod_pagespeed again this weekend when the traffic is low and see what happens.

OBTW, here is a list of the current modules enabled:

server:/var/www/includes# cd /etc/apache2/mods*en*
server::/etc/apache2/mods-enabled# ls -l
total 4
lrwxrwxrwx 1 root root  36 Sep  5  2018 access_compat.load -> ../mods-available/access_compat.load
lrwxrwxrwx 1 root root  28 Sep  5  2018 alias.conf -> ../mods-available/alias.conf
lrwxrwxrwx 1 root root  28 Sep  5  2018 alias.load -> ../mods-available/alias.load
lrwxrwxrwx 1 root root  33 Sep  5  2018 auth_basic.load -> ../mods-available/auth_basic.load
lrwxrwxrwx 1 root root  33 Sep  5  2018 authn_core.load -> ../mods-available/authn_core.load
lrwxrwxrwx 1 root root  33 Sep  5  2018 authn_file.load -> ../mods-available/authn_file.load
lrwxrwxrwx 1 root root  33 Sep  5  2018 authz_core.load -> ../mods-available/authz_core.load
lrwxrwxrwx 1 root root  33 Sep  5  2018 authz_host.load -> ../mods-available/authz_host.load
lrwxrwxrwx 1 root root  33 Sep  5  2018 authz_user.load -> ../mods-available/authz_user.load
lrwxrwxrwx 1 root root  32 Sep  5  2018 autoindex.conf -> ../mods-available/autoindex.conf
lrwxrwxrwx 1 root root  32 Sep  5  2018 autoindex.load -> ../mods-available/autoindex.load
lrwxrwxrwx 1 root root  30 Sep  5  2018 deflate.conf -> ../mods-available/deflate.conf
lrwxrwxrwx 1 root root  30 Sep  5  2018 deflate.load -> ../mods-available/deflate.load
lrwxrwxrwx 1 root root  26 Sep  5  2018 dir.conf -> ../mods-available/dir.conf
lrwxrwxrwx 1 root root  26 Sep  5  2018 dir.load -> ../mods-available/dir.load
lrwxrwxrwx 1 root root  26 Sep  5  2018 env.load -> ../mods-available/env.load
lrwxrwxrwx 1 root root  29 Sep  5  2018 filter.load -> ../mods-available/filter.load
-rw-r--r-- 1 root root 133 Dec 28  2018 geoip.conf
lrwxrwxrwx 1 root root  28 Sep  5  2018 geoip.load -> ../mods-available/geoip.load
lrwxrwxrwx 1 root root  30 Feb 22  2019 headers.load -> ../mods-available/headers.load
lrwxrwxrwx 1 root root  27 Sep  5  2018 mime.conf -> ../mods-available/mime.conf
lrwxrwxrwx 1 root root  27 Sep  5  2018 mime.load -> ../mods-available/mime.load
lrwxrwxrwx 1 root root  34 Sep  5  2018 mpm_prefork.conf -> ../mods-available/mpm_prefork.conf
lrwxrwxrwx 1 root root  34 Sep  5  2018 mpm_prefork.load -> ../mods-available/mpm_prefork.load
lrwxrwxrwx 1 root root  34 Sep  5  2018 negotiation.conf -> ../mods-available/negotiation.conf
lrwxrwxrwx 1 root root  34 Sep  5  2018 negotiation.load -> ../mods-available/negotiation.load
lrwxrwxrwx 1 root root  29 May  3 21:55 php5.6.conf -> ../mods-available/php5.6.conf
lrwxrwxrwx 1 root root  29 May  3 21:55 php5.6.load -> ../mods-available/php5.6.load
lrwxrwxrwx 1 root root  33 Sep  5  2018 reqtimeout.conf -> ../mods-available/reqtimeout.conf
lrwxrwxrwx 1 root root  33 Sep  5  2018 reqtimeout.load -> ../mods-available/reqtimeout.load
lrwxrwxrwx 1 root root  30 Sep  6  2018 rewrite.load -> ../mods-available/rewrite.load
lrwxrwxrwx 1 root root  31 Sep  5  2018 setenvif.conf -> ../mods-available/setenvif.conf
lrwxrwxrwx 1 root root  31 Sep  5  2018 setenvif.load -> ../mods-available/setenvif.load
lrwxrwxrwx 1 root root  36 Sep  6  2018 socache_shmcb.load -> ../mods-available/socache_shmcb.load
lrwxrwxrwx 1 root root  26 Sep  6  2018 ssl.conf -> ../mods-available/ssl.conf
lrwxrwxrwx 1 root root  26 Sep  6  2018 ssl.load -> ../mods-available/ssl.load
lrwxrwxrwx 1 root root  29 Sep  5  2018 status.conf -> ../mods-available/status.conf
lrwxrwxrwx 1 root root  29 Sep  5  2018 status.load -> ../mods-available/status.load

Neo · September 11, 2019, 2:23am

Also, it does look like I need to run the site though the WC3 HTML validator again as well.

Neo · September 11, 2019, 4:59am

OK. I think we are "more than OK" on WC3 HTML validator on the desktop (just fixed some pages).

But honestly, all of this had really nothing to do (that I can think of) with the title of this post:

"Your site has been switched to Mobile First Indexing"

and why a site like ours with over ninety percent all traffic from the web has been switched to Mobile First Indexing by Google; and what we should really do, if anything about it. I have already read the "google learn more" pages.

Neo · September 11, 2019, 6:42am

OK.... I ran the main pages though the validator and all looks fine ( .... ). Was some errors (most were warning before and are not listed as errors), so I fixed them (the errors, not the warnings).

OK.... Done... mod_pagespeed is enabled and verified running.

The last time I checked, our main pages were all HTTP/2 compliant in 2018. Maybe I missed something.

If you see any non-compliant pages, please let me know.

Thanks Akshay!

Neo · September 11, 2019, 7:07am

TURNED OFF FOR NOW - BREAKS THINGS

OK... testing this non-default mod_pagespeed config now:

    # Explicitly enables specific filters. This is useful in
    # conjuction with ModPagespeedRewriteLevel. For instance, filters
    # not included in the CoreFilters may be enabled using this
    # directive. This directive contains a comma-separated list of
    # filter names, and can be repeated.
    #
    ModPagespeedEnableFilters rewrite_javascript,rewrite_css
    ModPagespeedEnableFilters collapse_whitespace,elide_attributes

Neo · September 11, 2019, 8:01am

Now I recall some of the problems I used to have with mod_pagespeed ......

mod_pagespeed give a tiny increase in page speed with the big cost of caching and when we make a change on the site and try to clear the cache, the cache does not clear propertly or correctly
and it is very hard to work on the site, debug things, etc. due to the caching of all this code and the problems trying to clear the various caches.

So, I turned it off (again) for now.

I tried to clear the cache and it will not clear easily but again, it may be because we have to clear the cache on both the server side and the client side; all of which is too much work.

As I always try to explain to people over the years; all the "optimizations" come at a "cost" and the major cost for mod_pagespeed is a cache of a written pages (HTML, CSS, Javascript) which can be very difficult to manage if you are making any changes to the code or debugging something; and I started making changes to the code based on the W3C validator; but it's hard to work on this with mod_pagespee d caching, rewriting code, etc.

Anyway, its off again ... because the pages load fast already and mod_pagespeed does not really help that much and has very little or nothing to do with the issue I posted about, which is:

Your site has been switched to Mobile First Indexing by Google.

Cheers!

Neo · September 11, 2019, 8:52am

Well, not to give up ... turned it back on again and trying:

    # Explicitly enables specific filters. This is useful in
    # conjuction with ModPagespeedRewriteLevel. For instance, filters
    # not included in the CoreFilters may be enabled using this
    # directive. This directive contains a comma-separated list of
    # filter names, and can be repeated.
    #
    #ModPagespeedEnableFilters rewrite_javascript,rewrite_css
    ModPagespeedEnableFilters collapse_whitespace,elide_attributes

Will work on clearing the cache later

Neo · September 11, 2019, 8:59am

OK... instead of following the mod_pagespeed instructions for deleting the cache which seems not to work (at least for me), I just did this:

server:/etc/apache2/mods-enabled# rm -rf /var/cache/mod_pagespeed
server:/etc/apache2/mods-enabled# mkdir /var/cache/mod_pagespeed
server:/etc/apache2/mods-enabled# chown www-data.www-data  /var/cache/mod_pagespeed
server:/etc/apache2/mods-enabled# service apache2 restart

Akshay_Hegde · September 12, 2019, 3:09am

This is what I use, here is conf, I use query parameters whenever I change script/css, pagespeed re-creates cache

Example : somescript.js?v=20190912124147

ModPagespeedHonorCsp on
ModPagespeedForceCaching on
ModPagespeedRewriteLevel PassThrough

# limits
ModPagespeedFileCacheCleanIntervalMs 3600000
ModPagespeedFileCacheSizeKb 10240000
ModPagespeedFileCacheInodeLimit      500000
ModPagespeedLRUCacheKbPerProcess   2048
ModPagespeedLRUCacheByteLimit      32384

# failure fix
ModPagespeedRewriteUncacheableResources on

# disable filter via querystring
ModPagespeedForbidAllDisabledFilters true

# segment
ModPagespeedMaxSegmentLength 1800

# for css
ModPagespeedEnableFilters extend_cache_css
ModPagespeedMaxCombinedCssBytes -1
ModPagespeedEnableFilters move_css_to_head,move_css_above_scripts,rewrite_css,combine_css,outline_css

# for javascript
ModPagespeedEnableFilters extend_cache_scripts
ModPagespeedEnableFilters rewrite_javascript,rewrite_javascript_external,combine_javascript,outline_javascript
ModPagespeedJsOutlineMinBytes 1
ModPagespeedMaxCombinedJsBytes -1

#  fonts & others
ModPagespeedEnableFilters inline_google_font_css
ModPagespeedFetchHttps enable
ModPagespeedGoogleFontCssInlineMaxBytes -1
ModPagespeedCriticalImagesBeaconEnabled false

# html specific
ModPagespeedEnableFilters collapse_whitespace,pedantic,remove_comments

# image related
# default 8 images will be optimized
# -1 for unlimited
ModPagespeedImageMaxRewritesAtOnce -1

# Image optimize
ModPagespeedEnableFilters extend_cache_images
ModPagespeedEnableFilters convert_jpeg_to_progressive
ModPagespeedEnableFilters convert_png_to_jpeg

# for animated gif webp is not supporing everywhere
ModPagespeedEnableFilters  convert_jpeg_to_webp,convert_to_webp_lossless,convert_to_webp_animated

# at then end recompress all
ModPagespeedEnableFilters strip_image_meta_data,recompress_jpeg,recompress_png,recompress_webp

# dns prefetch in head
ModPagespeedEnableFilters insert_dns_prefetch

Neo · September 12, 2019, 3:28am

Yeah, that's what I do generally for Javascript and CSS. It's a bit more tricky when the change is to the HTML in a document, for me at least.

I will replace our pagespeed.conf file with the snippet you provided above and see how that works.

Do you mind to send me your entire pagespeed.conf file?

Thanks!

Akshay_Hegde · September 12, 2019, 4:28am

Ok I am attaching, many parameters are commented thats why I posted only parameters which I override at the end.

Neo · September 12, 2019, 4:37am

Thanks. I just wanted to see the different with the overall conf files in general; but I agree the first list you sent was the core changes to consider adding to the file, after looking at the overall files and comparing.

Will look more closely this weekend as our datacenter provider seems to be having some network issues today and so will delay working on changes until the datacenter network is more stable.

Thanks again, Akshay!

Neo · September 13, 2019, 3:53am

After further examination of mod_pageseed core filters , I'm leaving the mod_pagespeed configuration "as is" for now.

Instead, further to the topic of "Your site has been switched to Mobile First Indexing", I have added the forum stats to the home page on mobile:

Mobile: Advanced Forum Statistics to Forum Home Page

In the future, I think it's best to focus on SEO for mobile, since Google has switched out site to "mobile first indexing", so I think it is a good idea to put our most recently posts and most popular threads right on the home page.