Posts
Saturday, December 30, 2017
Just a reminder - if access is blocked to http://cefresearch.ca
There have been some recent messages about blocked access to the CEFSG sites. This most likely relates back to the Great Bot Attack of last December and January, and the need to block some of the IP addresses.
There is a notice on this blog here:
http://cefresearch.blogspot.ca/2017/03/we-are-stable.html
That explains how to retrieve you IP and send it to me to fix any problems.
Please bookmark the blog address and check in whenever you have a problem with the main site. The blog is on a different system and is not affected by the IP blocks.
http://cefresearch.blogspot.ca
Best wishes to all for 2018!
I will post a copy of this message on the forum as well.
Richard
Wednesday, November 22, 2017
CEFSG System Backups November 22, 2017
A complete backup of our 3 main systems was made this morning:
- Wordpress site: http://cefresearch.ca
- Main Forum: http://cefresearch.ca/phpBB3
This was completed prior to upgrading the Wordpress site to 4.9.
Sooner or later we are going to need to upgrade past phpBB3, I dread the day! At that time, we have to switch to a new system of pHp that is not directly upgradable.
If anyone notices any issues after today, please advise.
Remember also that at any time there are issues, a posting is made to our blog. The web or forum can be down, but you can still access the blog as it is a completely separate system. I believe in redundancy!
Thursday, June 8, 2017
CEFSG System Backups June 8, 2017
All CEFSG Systems received a FULL BACKUP this morning. I had near heart failure when I saw the message that there was a "[i]Maria Database Error - In Read Mode Only[/i]". I immediately went into the system and backed up the Forum, Wiki and Wordpress sites. Each of those are backed up using both the internal program backup system and [url=https://mysql.dfw1-2.websitesettings.com/?server=mysql51-021.wc2]phpMyAdmin[/url].
As it turned out, it appears it was a temporary "burp" at the server and the site(s) were fine. By the time I had completed the backups, everything was back working perfectly - so it appears.
Please let me know ASAP if anyone sees any issues on any of the three (3) sites:
Please remember (and bookmark) our blog site at http://cefresearch.blogspot.ca as that is where any notifications are posted if there are ever any site problems.
As it turned out, it appears it was a temporary "burp" at the server and the site(s) were fine. By the time I had completed the backups, everything was back working perfectly - so it appears.
Please let me know ASAP if anyone sees any issues on any of the three (3) sites:
Please remember (and bookmark) our blog site at http://cefresearch.blogspot.ca as that is where any notifications are posted if there are ever any site problems.
Tuesday, March 14, 2017
We are Stable!
All appears to be well and we have reactivated the Google Bot and if there are no problems we will allow the Bing Bot as well.
Remember to ALWAYS come to this Blog if you have problems with the CEFSG site, as when it is down or not available to you as an individual, we post the notice here.
For those that they find they do not have access, remember that we have a number of IP addresses blocked. To OPEN yours, you have to send me your IP address, User Name and local geographic address - as we have a number of international users. There are many sites that provide this - for example:
https://whatismyipaddress.com/
http://whoisip.ovh
Once I received that and confirm that you are a member, the IP address is cleared.
If you have a DYNAMIC IP instead of a STATIC IP you may have to file a notice again until we narrow down your area.
Remember to ALWAYS come to this Blog if you have problems with the CEFSG site, as when it is down or not available to you as an individual, we post the notice here.
For those that they find they do not have access, remember that we have a number of IP addresses blocked. To OPEN yours, you have to send me your IP address, User Name and local geographic address - as we have a number of international users. There are many sites that provide this - for example:
https://whatismyipaddress.com/
http://whoisip.ovh
Once I received that and confirm that you are a member, the IP address is cleared.
If you have a DYNAMIC IP instead of a STATIC IP you may have to file a notice again until we narrow down your area.
Monday, January 9, 2017
Update of the Great Bot Hunter for January 9, 2017
The .htaccess file has been changed to remove blocks on members that have reported in, plus the addition of new blocks on a number of new sites from Eastern Europe and Russia.
I have not repeated the list of blocks here but if anyone wants it, just ask. If you do not have access please remember to send me an e-mail to:
cefmatrix@gmail.com
Two other major steps were completed today:
I have not repeated the list of blocks here but if anyone wants it, just ask. If you do not have access please remember to send me an e-mail to:
cefmatrix@gmail.com
Two other major steps were completed today:
- The posts on the MAIN FORUM that deal with this issue have been moved to a MEMBERS ONLY section of the web site. That means that if you are not a member you do not even know that area of the forum exists.
- The CEFSG WIKI has been under increased attacks since the FORUM blocks were put in place so that is the one that is now PASSWORD protected. You need to enter both the correct USER NAME and PASSWORD to get into the Wiki. It has now been accessed by a member so I can confirm it is working - yahoo! That is not a simple process. Details on how to gain access is provided in the new MEMBERS ONLY section, as we don't want to publish that here on an open forum.
The size of the log file was significantly reduced today, which I take as a good sign. The really BAD BOT "SEMrushBOT" is not going down without a fight as it made 18,746 attempts to get into the site yesterday - all of which were successfully blocked. There were 36,582 entries in total, many of which were Google and Bing bots that are now rejected as well. Basically, if you are a BOT then "BUG OFF"for now and we will reconsider you later. GOOGLE and BING are "good bots" but for now we have to keep everyone away until we get this under control.
Lastly, we have had some very generous donations to the RESERVE FUND so I extend my thanks to all involved. As requested at the start of the DONATIONS program, the names of the donors are not identified publicly. Each donor "should" be receiving an automated confirmation e-mail once a donation is received.
Hopefully all of this will come to an end soon, it has been an annoyance to say the least!
Richard
Sunday, January 8, 2017
Changes for January 8, 2017
The .htaccess file has been changed to include these items:
# from Register.ca for Semrushbot 46.229. and then RVL added many from log of 6 January 2017 using Excel spreadsheet to sort into large groups
# bingbot is 157.55.39.104 and then added just 157.55.39. as many others used; 207.46.13.10
# mj12bot found in large group from 163.172.68.136 - on Sunday 8th made the block on 163.172.68. to catch all the rest (drop the .68.136 at end); also 46.4.32.75
# Domain Re-Animator Bot 167.114.156.198
# BoogleBot is 52.90.230.103 - very large amount on 6 January 2017 from IP in Seattle USA
# profound.net/domainappender trying to read .htaccess file 54.147.153.234
# downloading wiki from London UK 62.210.148.247; France 91.121.97.49
# large downloads from this Windson IP so temp block 70.51.99.113
# blocked any large groups from China, Russia, etc. also large group from Germany - Russia 5.255.250.66; Kiev 82.193.100.107; 91.200.
# Amsterdam 46.229.
# Sunday 8th added a large block of easter europe all in the 178. range; China 180.
# Sunday a very large download in Edmonton from 50.71.222.81 - system files
# list Google sites here: 66.249.73.142 - that did stop earlier as 66. was blocked, block 66.249 in case need to open 66.
# list Yahoo sites here: 68.180.228.157
# xpymep.exe at 146.158.83.16; 90.154.70.102
# THESE ONES HAVE BEEN OPENED AFTER THE INITIAL BLOCK WAS PUT IN PLACE
# 67 (Ted Walshe, Montreal & Neil Burns); 69 (Diane Johnson Brantford - changed Sunday new IP); 124 (Mark, Australia)
Require all granted
Require not ip 10.8.163.19
Require not ip 10.8.174.151
Require not ip 104.140.
Require not ip 2.
Require not ip 5.196.167.230
Require not ip 5.9.94.207
Require not ip 5.255.250.66
Require not ip 10.
Require not ip 10.8.
Require not ip 11.
Require not ip 12.
Require not ip 13.
Require not ip 14.
Require not ip 15.
Require not ip 16.
Require not ip 17.
Require not ip 18.
Require not ip 19.
Require not ip 46.
Require not ip 46.229.
Require not ip 50.71.222.81
Require not ip 52.90.230.103
Require not ip 54.147.153.234
Require not ip 61.
Require not ip 62.
Require not ip 62.210.148.247
Require not ip 63.
Require not ip 64.
Require not ip 65.
Require not ip 66.
Require not ip 66.249.
Require not ip 66.249.73.142
Require not ip 68.
Require not ip 68.180.
Require not ip 70.51.99.113
Require not ip 77.248.252.113
Require not ip 82.193.
Require not ip 90.154.70.102
Require not ip 91.121.
Require not ip 91.200.
Require not ip 104.
Require not ip 104.140.
Require not ip 106.
Require not ip 107.
Require not ip 108.
Require not ip 109.
Require not ip 110.
Require not ip 111.
Require not ip 112.
Require not ip 113.
Require not ip 114.
Require not ip 115.
Require not ip 116.
Require not ip 117.
Require not ip 118.
Require not ip 119.
Require not ip 120.
Require not ip 121.
Require not ip 122.
Require not ip 123.
Require not ip 125.
Require not ip 126.
Require not ip 127.
Require not ip 128.
Require not ip 129.
Require not ip 133.130
Require not ip 144.
Require not ip 146.
Require not ip 151.237.
Require not ip 151.80.
Require not ip 154.20.7.7
Require not ip 157.55.39.104
Require not ip 157.55.39.
Require not ip 163.172.
Require not ip 165.231.
Require not ip 167.114.156.198
Require not ip 173.234.159.250
Require not ip 178.
Require not ip 180.
Require not ip 207.46.13.10
Require not ip 207.81.234.223
As well all these bots are blocked. The first line is supposed to block all bots with the "*" but many seem to get around this process:
User-agent: *
Disallow: /
User-agent: mj12bot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: ChinasoSpider
Disallow: /
User-agent: ToutiaoSpider
Disallow: /
User-agent: Yahoo Slurp
Disallow: /
User-agent: YahooCacheSystem
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: bingbot
Disallow: /
User-agent: EasouSpider
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: YYSpider
Disallow: /
User-Agent: YoudaoBot
Disallow: /
User-Agent:360Spider
Disallow:/
User-Agent:360Spider-Image
Disallow:/
User-Agent: sogou spider
Disallow: /
User-Agent: Sogou web spider
Disallow: /
User-Agent: MJ12bot
Disallow:/
user-agent: AhrefsBot
disallow: /
user-agent: YRSpider
disallow: /
user-agent: 360spider-image
disallow: /
# from Register.ca for Semrushbot 46.229. and then RVL added many from log of 6 January 2017 using Excel spreadsheet to sort into large groups
# bingbot is 157.55.39.104 and then added just 157.55.39. as many others used; 207.46.13.10
# mj12bot found in large group from 163.172.68.136 - on Sunday 8th made the block on 163.172.68. to catch all the rest (drop the .68.136 at end); also 46.4.32.75
# Domain Re-Animator Bot 167.114.156.198
# BoogleBot is 52.90.230.103 - very large amount on 6 January 2017 from IP in Seattle USA
# profound.net/domainappender trying to read .htaccess file 54.147.153.234
# downloading wiki from London UK 62.210.148.247; France 91.121.97.49
# large downloads from this Windson IP so temp block 70.51.99.113
# blocked any large groups from China, Russia, etc. also large group from Germany - Russia 5.255.250.66; Kiev 82.193.100.107; 91.200.
# Amsterdam 46.229.
# Sunday 8th added a large block of easter europe all in the 178. range; China 180.
# Sunday a very large download in Edmonton from 50.71.222.81 - system files
# list Google sites here: 66.249.73.142 - that did stop earlier as 66. was blocked, block 66.249 in case need to open 66.
# list Yahoo sites here: 68.180.228.157
# xpymep.exe at 146.158.83.16; 90.154.70.102
# THESE ONES HAVE BEEN OPENED AFTER THE INITIAL BLOCK WAS PUT IN PLACE
# 67 (Ted Walshe, Montreal & Neil Burns); 69 (Diane Johnson Brantford - changed Sunday new IP); 124 (Mark, Australia)
Require all granted
Require not ip 10.8.163.19
Require not ip 10.8.174.151
Require not ip 104.140.
Require not ip 2.
Require not ip 5.196.167.230
Require not ip 5.9.94.207
Require not ip 5.255.250.66
Require not ip 10.
Require not ip 10.8.
Require not ip 11.
Require not ip 12.
Require not ip 13.
Require not ip 14.
Require not ip 15.
Require not ip 16.
Require not ip 17.
Require not ip 18.
Require not ip 19.
Require not ip 46.
Require not ip 46.229.
Require not ip 50.71.222.81
Require not ip 52.90.230.103
Require not ip 54.147.153.234
Require not ip 61.
Require not ip 62.
Require not ip 62.210.148.247
Require not ip 63.
Require not ip 64.
Require not ip 65.
Require not ip 66.
Require not ip 66.249.
Require not ip 66.249.73.142
Require not ip 68.
Require not ip 68.180.
Require not ip 70.51.99.113
Require not ip 77.248.252.113
Require not ip 82.193.
Require not ip 90.154.70.102
Require not ip 91.121.
Require not ip 91.200.
Require not ip 104.
Require not ip 104.140.
Require not ip 106.
Require not ip 107.
Require not ip 108.
Require not ip 109.
Require not ip 110.
Require not ip 111.
Require not ip 112.
Require not ip 113.
Require not ip 114.
Require not ip 115.
Require not ip 116.
Require not ip 117.
Require not ip 118.
Require not ip 119.
Require not ip 120.
Require not ip 121.
Require not ip 122.
Require not ip 123.
Require not ip 125.
Require not ip 126.
Require not ip 127.
Require not ip 128.
Require not ip 129.
Require not ip 133.130
Require not ip 144.
Require not ip 146.
Require not ip 151.237.
Require not ip 151.80.
Require not ip 154.20.7.7
Require not ip 157.55.39.104
Require not ip 157.55.39.
Require not ip 163.172.
Require not ip 165.231.
Require not ip 167.114.156.198
Require not ip 173.234.159.250
Require not ip 178.
Require not ip 180.
Require not ip 207.46.13.10
Require not ip 207.81.234.223
As well all these bots are blocked. The first line is supposed to block all bots with the "*" but many seem to get around this process:
User-agent: *
Disallow: /
User-agent: mj12bot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: ChinasoSpider
Disallow: /
User-agent: ToutiaoSpider
Disallow: /
User-agent: Yahoo Slurp
Disallow: /
User-agent: YahooCacheSystem
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: bingbot
Disallow: /
User-agent: EasouSpider
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: YYSpider
Disallow: /
User-Agent: YoudaoBot
Disallow: /
User-Agent:360Spider
Disallow:/
User-Agent:360Spider-Image
Disallow:/
User-Agent: sogou spider
Disallow: /
User-Agent: Sogou web spider
Disallow: /
User-Agent: MJ12bot
Disallow:/
user-agent: AhrefsBot
disallow: /
user-agent: YRSpider
disallow: /
user-agent: 360spider-image
disallow: /
Saturday, January 7, 2017
SYSTEM CHANGES: Saturday January 7, 2017 $$$$
Unfortunately the BOTS have not only continued to ATTACK but they are growing in number.
Changes made to date have significantly reduced our "compute cycles" but we are still way over the limit and the costs are increasing daily. You can see the effect the changes made to date have made in the graph:
https://www.mediafire.com/convkey/87ef/i1r78skw21rv1kb6g.jpg
Our compute cycles have dropped from 2500 a day to less than 600 a day and our bandwidth has dropped from 4000 to less than 800. That is from blocking some of the worst offenders.
The PROBLEM is that we are only allowed 3000 compute cycles for the WHOLE MONTH period, which runs from the 22nd of Month A to 21st of Month B. Additional charges for the period 16 November to 15 December were $360.26 so there will be another large charge for December-January. We normally pay less than that for the whole year!
A few of you know that I was making changes to increase the BLOT BLOCKAGE today, as I made a typo and the server crashed. That was quickly fixed and from that point on I tested out all the files on a blank directory before they are set to load on the site here. That work is all done and the files are going to be loaded shortly. I do check to see that there are not too many people on the system when I make that change but unfortunately I can not account for someone who is logging on at the exact same time the files are uploading.
The new .htaccess file that carries all the new blocking code will be blocking a large number if IP addresses and ranges of IP addresses associated with what appear to be the main offenders. We have tried other less severe approaches and the BOTS are bypassing them easily. Neil found a way to just say "NO BOTS PLEASE" and they continued the attack. It could be it takes a few days but at the rate the costs are increasing we have to take every step possible.
IF YOU FIND that you are blocked, then I hope that you remembered to bookmark the site where the announcements are posted here:
http://cefresearch.blogspot.ca/
as I will post a copy of this message at that location. I will put in the e-mail address for the MATRIX in case you don't have that, then you can send me a message with your I.P. address so I can exempt that block.
Find your IP address: http://whoisip.ovh
I will also provide a list of blocked addresses to date at that location. Doing that here would just let the BOTS know who is being blocked.
I also need to you to tell me if any of you are using these IP addresses as they appear somewhat normal, of origin in the USA or Canada, but are associated with what appear to be large data downloads:
Halifax 142.177.247.89
Winnipeg 142.161.238.42
Ottawa 131.137.88.71
Melbourne Australia 124.191.103.90 (might be the Diggers)
Saskatoon 128.233.6.93
Redmond Kansas 131.253.24.140
Calgary 137.186.55.136
Tatamagouche (N.S. or P.E.I?) 142.134.66.130
Brantford 50.100.61.223
Edmonton 50.64.149.31 or 50.68.152.77
Windsor 70.51.99.113 (that is now blocked as it does not appear to be HUMAN - if you are call home!)
I will post this now and come back at 3:00 pm EST to upload the new blocks. You might want to finish any posts you are working on now prior to that time, JUST IN CASE! It should work cleanly but better safe than sorry.
Stay tuned to the other announcement site over the next week or until further notice, as if this keeps up and the blocks DON'T WORK then we will probably need to block the site, or we will go broke.
I have already sent in a submission to a VPS PROVIDER (http://vpsville.ca/) to determine what is involved (and costs) to switch to a VPS (Virtual Private Server) system. I know nothing about those, other than what I have read, and I do not know if I would have the necessary skill set or time to operate such a system. I have not heard back from this as of this date. Reading tells me there are MANAGED VPS and STAND ALONE VPS and so if we can get a MANAGED system, it might be okay. Best is if we can stay where we are now if we can beat the botters to death!
Fingers crossed, now also crossing toes,
Richard
ADDED TO THIS BLOG POST ONLY
Here is "section" the file that is now on the system to do the blocking. If you see one that is a short version such as "Require not ip 124." that means that it is blocking any IP that starts with that series of numbers. If your IP has any of the front numbers on the list you have to let me know by e-mail to cefmatrix@gmail.com.
# from Register.ca for Semrushbot 46.229. and then RVL added many from log of 6 January 2017 using Excel spreadsheet to sort into large groups
# bingbot is 157.55.39.104 and then added just 157.55.39. as many others used
# mj12bot found in large group from 163.172.68.136
# Domain Re-Animator Bot 167.114.156.198
# BoogleBot is 52.90.230.103 - very large amount on 6 January 2017 from IP in Seattle USA
# profound.net/domainappender trying to read .htaccess file 54.147.153.234
# downloading wiki from London UK 62.210.148.247
# large downloads from this Windson IP so temp block 70.51.99.113
# blocked any large groups from China, Russia, etc. also large group from Germany
Require all granted
Require not ip 46.229.
Require not ip 10.8.163.19
Require not ip 10.8.174.151
Require not ip 10.8.
Require not ip 104.140.
Require not ip 107.
Require not ip 108.
Require not ip 109.
Require not ip 110.
Require not ip 111.
Require not ip 112.
Require not ip 113.
Require not ip 114.
Require not ip 115.
Require not ip 116.
Require not ip 117.
Require not ip 118.
Require not ip 119.
Require not ip 12.106.
Require not ip 120.
Require not ip 121.
Require not ip 122.
Require not ip 123.
Require not ip 124.
Require not ip 125.
Require not ip 126.
Require not ip 127.
Require not ip 128.
Require not ip 129.
Require not ip 144.
Require not ip 146.
Require not ip 151.237.
Require not ip 151.80
Require not ip 157.55.39.104
Require not ip 157.55.39.
Require not ip 163.172.68.136
Require not ip 165.231.
Require not ip 167.114.156.198
Require not ip 17.
Require not ip 18.
Require not ip 5.196.167.230
Require not ip 5.9.94.207
Require not ip 52.90.230.103
Require not ip 54.147.153.234
Require not ip 61.
Require not ip 62.
Require not ip 62.210.148.247
Require not ip 63.
Require not ip 64.
Require not ip 65.
Require not ip 66.
Require not ip 67.
Require not ip 68.
Require not ip 69.
Require not ip 70.51.99.113
Require not ip 77.248.252.113
This may be duplication, but is also added to the .htaccess code as some can ignore the IP address blocks. These are the NAMES of the ones that I have found so far that are the main offenders:
# RVL added December 31, 2016 as Semrushbot is still invading the site
# found here: http://stackoverflow.com/questions/23631872/ban-robots-from-website
# they say "Also you can do this little trick, deny ANY ip address that has "SemrushBot" in user agent string"
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
SetEnvIfNoCase User-Agent "^SemrushBot" bad_user
SetEnvIfNoCase User-Agent "^Slurp" bad_user
SetEnvIfNoCase User-Agent "^dotbot" bad_user
SetEnvIfNoCase User-Agent "^Googlebot" bad_user
SetEnvIfNoCase User-Agent "^bingbot" bad_user
SetEnvIfNoCase User-Agent "^mj12bot" bad_user
SetEnvIfNoCase User-Agent "^ToutiaoSpider" bad_user
SetEnvIfNoCase User-Agent "^YandexBot" bad_user
SetEnvIfNoCase User-Agent "^domainappender" bad_user
SetEnvIfNoCase User-Agent "^bogglebot" bad_user
SetEnvIfNoCase User-Agent "^WhateverElseBadUserAgentHere" bad_user
Deny from env=bad_user
In the event that you understand any of this, then I can tell you that the following code was also added to CACHE the files on the site so they are not downloaded from the server every time someone visits a post:
# ADDED from http://www.fastcomet.com/tutorials/phpbb3/performance-optimization 28-12-2016
## EXPIRES CACHING ##
ExpiresActive On
ExpiresByType image/jpg "access plus 1 year"
ExpiresByType image/jpeg "access plus 1 year"
ExpiresByType image/gif "access plus 1 year"
ExpiresByType image/png "access plus 1 year"
ExpiresByType text/css "access plus 1 month"
ExpiresByType application/pdf "access plus 1 month"
ExpiresByType text/x-javascript "access plus 1 month"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
ExpiresByType image/x-icon "access plus 1 year"
ExpiresDefault "access plus 2 days"
## EXPIRES CACHING ##
If you think I actually know what I am doing, think again! I go read about it on the web, try it out, and then sit back to see if it works. If anyone really knows how to do all this stuff, then you could be a big asset to the team! If you see line of code that starts with a # that means that line is just a note and is not read as code. There you will see I put in the URL of the information that I may have found on the web that tells me what I should do to fix the problem. Very often I have no idea what they are talking about so I have to learn that first.
Added later:
The file was uploaded at 3:02 pm EST and the site is still alive! Whew!
The blocks are now in effect.
Changes made to date have significantly reduced our "compute cycles" but we are still way over the limit and the costs are increasing daily. You can see the effect the changes made to date have made in the graph:
https://www.mediafire.com/convkey/87ef/i1r78skw21rv1kb6g.jpg
Our compute cycles have dropped from 2500 a day to less than 600 a day and our bandwidth has dropped from 4000 to less than 800. That is from blocking some of the worst offenders.
The PROBLEM is that we are only allowed 3000 compute cycles for the WHOLE MONTH period, which runs from the 22nd of Month A to 21st of Month B. Additional charges for the period 16 November to 15 December were $360.26 so there will be another large charge for December-January. We normally pay less than that for the whole year!
A few of you know that I was making changes to increase the BLOT BLOCKAGE today, as I made a typo and the server crashed. That was quickly fixed and from that point on I tested out all the files on a blank directory before they are set to load on the site here. That work is all done and the files are going to be loaded shortly. I do check to see that there are not too many people on the system when I make that change but unfortunately I can not account for someone who is logging on at the exact same time the files are uploading.
The new .htaccess file that carries all the new blocking code will be blocking a large number if IP addresses and ranges of IP addresses associated with what appear to be the main offenders. We have tried other less severe approaches and the BOTS are bypassing them easily. Neil found a way to just say "NO BOTS PLEASE" and they continued the attack. It could be it takes a few days but at the rate the costs are increasing we have to take every step possible.
IF YOU FIND that you are blocked, then I hope that you remembered to bookmark the site where the announcements are posted here:
http://cefresearch.blogspot.ca/
as I will post a copy of this message at that location. I will put in the e-mail address for the MATRIX in case you don't have that, then you can send me a message with your I.P. address so I can exempt that block.
Find your IP address: http://whoisip.ovh
I will also provide a list of blocked addresses to date at that location. Doing that here would just let the BOTS know who is being blocked.
I also need to you to tell me if any of you are using these IP addresses as they appear somewhat normal, of origin in the USA or Canada, but are associated with what appear to be large data downloads:
Halifax 142.177.247.89
Winnipeg 142.161.238.42
Ottawa 131.137.88.71
Melbourne Australia 124.191.103.90 (might be the Diggers)
Saskatoon 128.233.6.93
Redmond Kansas 131.253.24.140
Calgary 137.186.55.136
Tatamagouche (N.S. or P.E.I?) 142.134.66.130
Brantford 50.100.61.223
Edmonton 50.64.149.31 or 50.68.152.77
Windsor 70.51.99.113 (that is now blocked as it does not appear to be HUMAN - if you are call home!)
I will post this now and come back at 3:00 pm EST to upload the new blocks. You might want to finish any posts you are working on now prior to that time, JUST IN CASE! It should work cleanly but better safe than sorry.
Stay tuned to the other announcement site over the next week or until further notice, as if this keeps up and the blocks DON'T WORK then we will probably need to block the site, or we will go broke.
I have already sent in a submission to a VPS PROVIDER (http://vpsville.ca/) to determine what is involved (and costs) to switch to a VPS (Virtual Private Server) system. I know nothing about those, other than what I have read, and I do not know if I would have the necessary skill set or time to operate such a system. I have not heard back from this as of this date. Reading tells me there are MANAGED VPS and STAND ALONE VPS and so if we can get a MANAGED system, it might be okay. Best is if we can stay where we are now if we can beat the botters to death!
Fingers crossed, now also crossing toes,
Richard
ADDED TO THIS BLOG POST ONLY
Here is "section" the file that is now on the system to do the blocking. If you see one that is a short version such as "Require not ip 124." that means that it is blocking any IP that starts with that series of numbers. If your IP has any of the front numbers on the list you have to let me know by e-mail to cefmatrix@gmail.com.
# from Register.ca for Semrushbot 46.229. and then RVL added many from log of 6 January 2017 using Excel spreadsheet to sort into large groups
# bingbot is 157.55.39.104 and then added just 157.55.39. as many others used
# mj12bot found in large group from 163.172.68.136
# Domain Re-Animator Bot 167.114.156.198
# BoogleBot is 52.90.230.103 - very large amount on 6 January 2017 from IP in Seattle USA
# profound.net/domainappender trying to read .htaccess file 54.147.153.234
# downloading wiki from London UK 62.210.148.247
# large downloads from this Windson IP so temp block 70.51.99.113
# blocked any large groups from China, Russia, etc. also large group from Germany
Require all granted
Require not ip 46.229.
Require not ip 10.8.163.19
Require not ip 10.8.174.151
Require not ip 10.8.
Require not ip 104.140.
Require not ip 107.
Require not ip 108.
Require not ip 109.
Require not ip 110.
Require not ip 111.
Require not ip 112.
Require not ip 113.
Require not ip 114.
Require not ip 115.
Require not ip 116.
Require not ip 117.
Require not ip 118.
Require not ip 119.
Require not ip 12.106.
Require not ip 120.
Require not ip 121.
Require not ip 122.
Require not ip 123.
Require not ip 124.
Require not ip 125.
Require not ip 126.
Require not ip 127.
Require not ip 128.
Require not ip 129.
Require not ip 144.
Require not ip 146.
Require not ip 151.237.
Require not ip 151.80
Require not ip 157.55.39.104
Require not ip 157.55.39.
Require not ip 163.172.68.136
Require not ip 165.231.
Require not ip 167.114.156.198
Require not ip 17.
Require not ip 18.
Require not ip 5.196.167.230
Require not ip 5.9.94.207
Require not ip 52.90.230.103
Require not ip 54.147.153.234
Require not ip 61.
Require not ip 62.
Require not ip 62.210.148.247
Require not ip 63.
Require not ip 64.
Require not ip 65.
Require not ip 66.
Require not ip 67.
Require not ip 68.
Require not ip 69.
Require not ip 70.51.99.113
Require not ip 77.248.252.113
This may be duplication, but is also added to the .htaccess code as some can ignore the IP address blocks. These are the NAMES of the ones that I have found so far that are the main offenders:
# RVL added December 31, 2016 as Semrushbot is still invading the site
# found here: http://stackoverflow.com/questions/23631872/ban-robots-from-website
# they say "Also you can do this little trick, deny ANY ip address that has "SemrushBot" in user agent string"
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
SetEnvIfNoCase User-Agent "^SemrushBot" bad_user
SetEnvIfNoCase User-Agent "^Slurp" bad_user
SetEnvIfNoCase User-Agent "^dotbot" bad_user
SetEnvIfNoCase User-Agent "^Googlebot" bad_user
SetEnvIfNoCase User-Agent "^bingbot" bad_user
SetEnvIfNoCase User-Agent "^mj12bot" bad_user
SetEnvIfNoCase User-Agent "^ToutiaoSpider" bad_user
SetEnvIfNoCase User-Agent "^YandexBot" bad_user
SetEnvIfNoCase User-Agent "^domainappender" bad_user
SetEnvIfNoCase User-Agent "^bogglebot" bad_user
SetEnvIfNoCase User-Agent "^WhateverElseBadUserAgentHere" bad_user
Deny from env=bad_user
In the event that you understand any of this, then I can tell you that the following code was also added to CACHE the files on the site so they are not downloaded from the server every time someone visits a post:
# ADDED from http://www.fastcomet.com/tutorials/phpbb3/performance-optimization 28-12-2016
## EXPIRES CACHING ##
ExpiresActive On
ExpiresByType image/jpg "access plus 1 year"
ExpiresByType image/jpeg "access plus 1 year"
ExpiresByType image/gif "access plus 1 year"
ExpiresByType image/png "access plus 1 year"
ExpiresByType text/css "access plus 1 month"
ExpiresByType application/pdf "access plus 1 month"
ExpiresByType text/x-javascript "access plus 1 month"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
ExpiresByType image/x-icon "access plus 1 year"
ExpiresDefault "access plus 2 days"
## EXPIRES CACHING ##
If you think I actually know what I am doing, think again! I go read about it on the web, try it out, and then sit back to see if it works. If anyone really knows how to do all this stuff, then you could be a big asset to the team! If you see line of code that starts with a # that means that line is just a note and is not read as code. There you will see I put in the URL of the information that I may have found on the web that tells me what I should do to fix the problem. Very often I have no idea what they are talking about so I have to learn that first.
Added later:
The file was uploaded at 3:02 pm EST and the site is still alive! Whew!
The blocks are now in effect.
Subscribe to:
Posts (Atom)