But there are things like public Search Engine that doesnt need user to be authenticated in order to find on Facebook a person via search or access Profile info page.
When I was coding in PHP and working with Prestashop e-commerce I have used a Crawler/Bot similar to the one in this gist, but this can be easily spoofed, because it's based on the HTTP_USER_AGENT. A better approach here is to use IP addresses to lookup the good know crawlers, aka the ones from searches engines like Google and Bing, but this will not work for keeping at bay the bad crawlers and bots. because they switch IP addresses very frequently.
But going through couple of organisations I did not manage to get any public api request that I could access via Postman or simple via url.
Companies like Facebook or even smaller ones, that have a good amount of resources at their disposal, use Artificial Intelligence(AI) to try to draw the line between Who is doing good and bad requests, and this type of software is known as User Behavior Analytics (UBA):
User behavior analytics (UBA) as defined by Gartner is a cybersecurity process about detection of insider threats, targeted attacks, and financial fraud. UBA solutions look at patterns of human behavior, and then apply algorithms and statistical analysis to detect meaningful anomalies from those patterns—anomalies that indicate potential threats. Instead of tracking devices or security events, UBA tracks a system's users. Big data platforms like Apache Hadoop are increasing UBA functionality by allowing them to analyze petabytes worth of data to detect insider threats and advanced persistent threats.
So this must be why you have such a difficulty in getting through the APIs of companies like Facebook, but that doesn't mean it's not possible, because Hackers have being do it, and the number of data breaches occurring every year in big companies are a proof of that.
The reason why I am asking this question, because it is so easy to find some MERN Stack course and they will teach you how to handle authentication for authorized users and so one. Or even the LAMP technologies. But no one explains how to protect this data without requiring any user to be signed in.
Well that may be because of a common misconception among developers that don't really understand the difference between Who vs What is doing the request to the API server.
The Difference Between WHO and WHAT is Accessing the API Server
I wrote a series of articles around API and Mobile security, and from the article Why Does Your Mobile App Need An Api Key? I will quote the following:
The what is the thing making the request to the API server. Is it really a genuine instance of your mobile app, or is it a bot, an automated script or an attacker manually poking around your API server with a tool like Postman?
The who is the user of the mobile app that we can authenticate, authorize and identify in several ways, like using OpenID Connect or OAUTH2 flows.
Think about the Who as the user your API server will be able to Authenticate and Authorize access to the data, and think about the What as the software making that request in behalf of the user.
So, in my opinion a lot of Developers are not aware of this difference between Who and What in a request, therefore they concentrate in solutions for the Who.
Possible Solutions
So how do I secure those above described public api routes in my backend application. And how does it do big organisations like Facebook, Google, LinkedIn and etc.
This organizations are using very sophisticated UBA solutions that may not be in the reach of every organization in terms of cost or because they are proprietary solutions, but other solutions exists, and you can read the section Defending the API Server on this other reply I gave to the question secure api data from calls out of the app to understand how you can gradually improve the security of an API server for a web app.
If you also need to secure the API server for requests from a mobile app, then you can lock it down with a very high degree of confidence to your mobile app, by employing the Mobile App Attestation concept, and you can read more about in this reply I gave to the question How to secure an API REST for mobile app?.
Do you Want to Go the Extra Mile?
I cannot finish any response to a security question without give a reference to the excellent work from the OWASP foundation.
For Web Apps
OWASP Web Top 10 Risks
The OWASP Top 10 is a powerful awareness document for web application security. It represents a broad consensus about the most critical security risks to web applications. Project members include a variety of security experts from around the world who have shared their expertise to produce this list.
The Web Security Testing Guide:
The OWASP Web Security Testing Guide includes a "best practice" penetration testing framework which users can implement in their own organizations and a "low level" penetration testing guide that describes techniques for testing most common web application and web service security issues.
For Mobile Apps
OWASP Mobile Security Project - Top 10 risks
The OWASP Mobile Security Project is a centralized resource intended to give developers and security teams the resources they need to build and maintain secure mobile applications. Through the project, our goal is to classify mobile security risks and provide developmental controls to reduce their impact or likelihood of exploitation.
OWASP - Mobile Security Testing Guide:
The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security development, testing and reverse engineering.
For APIS
OWASP API Security Top 10
The OWASP API Security Project seeks to provide value to software developers and security assessors by underscoring the potential risks in insecure APIs, and illustrating how these risks may be mitigated. In order to facilitate this goal, the OWASP API Security Project will create and maintain a Top 10 API Security Risks document, as well as a documentation portal for best practices when creating or assessing APIs.
