1

I'm setting up my web page to support ssr and here comes my question, can I know if the client is a web-crawler so I can do ssr?

This way I will serve my web-page as it is to clients that are not web-crawlers

I have seen that to verify google-bot-crawler you can use https://stackoverflow.com/a/3308728/8991228

But is there a general way of doing so?

miggy92
  • 70
  • 10

1 Answers1

1

There is a header: User-Agent and it is usually with the help of him that you are able to recognize whether it is a browser or a bot, but...

The difficulty of falsifying this header is 0.

Therefore, additional verification methods are used, e.g. Google, as you have shown.

But... Not all bots appear as bots. For example, Google has a tendency to check if another content is being sent to the bot.

In sum: You can do it if you know that the bot accepts it (eg for Facebook link sharer)

bato3
  • 2,695
  • 1
  • 18
  • 26
  • I think this could work. I just want to know whether to render my page on server or in client. Could it be that using `User-Agent` to render in server or client will result in some bad behavior caused by malicious clients as it is easy for falsifying? – miggy92 Mar 28 '19 at 15:22
  • You need to do an analysis that bad customer behavior will harm you and limit it. You can ignore the others. Answer the question: *Why should you be nice to bad guests?* BTW: you can transfer the page rendition to Google. It is doing quite well with JS. – bato3 Mar 29 '19 at 11:14
  • BTW: There is also a third group of recipients: web scraper, e.g. CURL, Guzzle... HTTP client for server side – bato3 Mar 29 '19 at 11:16
  • I think there is not general solution for this. There is a solution for every bot-crawler. But `User-Agent` is the most general solution I can think about. – miggy92 Mar 29 '19 at 15:55