Reporting Forum Error

[Cardinal51]

Probably not really interesting and I am not sure where to post.

But at a certain moment I got a ton of these, just surfing to the forum, yesterday:

Screenshot 2024-11-19 205419.jpg

I guess the whole TSC development team was trying to surf the forum at the same time :-)

And this just now using the search engine:

Screenshot 2024-11-20 191151.jpg

[buzz456]

For some unknown reason there was like 2000+ guests on here off and on yesterday. If you can ever figure out the internet please let me know.

[Cardinal51]

Checking the logs can help you figuring out they where real people or crawlers.

Do you give crawlers access to the forums fir indexing (and therefore make you show up in various searching) ?\

- - - a little later - -
Just check and seems the forum is not indexed. I strongly suggest to change that, would be great people can actually find the forum.

[buzz456]

I have no idea what you are talking about.

[AmericanSteam]

I had the same issue. Unregistered users taking up all the connections. Maybe time to lock it down to unregistered users and bots.

[Cardinal51]

Nah, let them in and index the forum. At least the famous ones like BANG / GEEGLO /. YOHAA

[buzz456]

I still have no idea what you are talking about. What is indexing the forum?

[Chacal]

Indexing the forums is done by "good" bots called crawlers. They discover our pages and add them to, for example, Google search data, so that people searching Google for, let's say "Bangor & Aroostook" will get some results from our site. In short it makes our content visible.

Yesterday's problem was caused by "bad" bots. They open lots of connections and don't close them, making the site and forums unavailable. The forum database has a maximum number of connections and refuses to open more.

I fixed the problem by analyzing the logs and blocking some IP ranges and some whole countries (China, Russia and Turkey) directly at the firewall. Now they can't even reach our site.

[AmericanSteam]

Indexing the forums is done by "good" bots called crawlers. They discover our pages and add them to, for example, Google search data, so that people searching Google for, let's say "Bangor & Aroostook" will get some results from our site. In short it makes our content visible.

Yesterday's problem was caused by "bad" bots. They open lots of connections and don't close them, making the site and forums unavailable. The forum database has a maximum number of connections and refuses to open more.

I fixed the problem by analyzing the logs and blocking some IP ranges and some whole countries (China, Russia and Turkey) directly at the firewall. Now they can't even reach our site.

Thank you for your efforts. Keep on fine tuning. I would think some of these "bad" actors are perusing anything with railroad in it to find information that could be used for nefarious means.
I see a lot of this in Facebook chat rooms here in Alaska. We are near Russia and the military has a big presence here as well as energy production. The administrators are vigilant in rooting out bad actors.

[Cardinal51]

Indexing the forums is done by "good" bots......

I think the forum is not indexed at all.

[Chacal]

I know, we used to allow a list of crawlers but the information is gone, I think there was a reason but I can't remember what it was.

I need to repopulate this list, that's somewhere in my pile of to-dos.

Edit: actually the forums should be indexed anyways, I shouldn't need to do anything.
The bots have guest access, they should see most of the forum posts.
Not sure why this wouldn't work.

[Cardinal51]

This is rhe problem .....

https://www.google.com/search?q=train+s ... merica.com

Screenshot_20241121-204049-edit-20241121204121.jpg

Perhaps your robots.txt is the culprit.

[Chacal]

Yes I was reading the very same page.
I have to check my robots.txt.

[Cardinal51]

It is great news that the forum has this issue! After the fix we wil have more visitors coming over. :-)

Have you considered giving human non- members/ aka people-not-logged-in reading rights for the first post of each thread ? (of course crawlers should have rights to read all of it)

[Chacal]

These are guests. They have read access to most of the forums and posts.

Also, as I expected, Googlebot was blocked explicitely. I have removed this block for now, and will monitor the impact on the forums and web site.