If I attempt accessing it with lynx, I get this generic error:
"Due to aggressive automated scraping of FederalRegister.gov and
eCFR.gov, programmatic access to these sites is limited to access to our extensive developer APIs."
I don't get this message on a normal javascript enabled browser. I
surmise it views queries from non-javascript browsers as a curl-like command.
On 02 Jul 2024, Daniel said the following...
If I attempt accessing it with lynx, I get this generic error:
"Due to aggressive automated scraping of FederalRegister.gov and eCFR.gov, programmatic access to these sites is limited to access to our extensive developer APIs."
I don't get this message on a normal javascript enabled browser. I surmise it views queries from non-javascript browsers as a curl-like command.
I don't know if you can change the user agent in lynx, but try and see if it works with curl.
e.g:
agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
curl -A "$agent" <url>
I've had to do this in the past when pulling information for weather, they stopped allowing curl, but simply changing the user agent allowed it to work.
Sysop: | digital man |
---|---|
Location: | Riverside County, California |
Users: | 1,036 |
Nodes: | 15 (1 / 14) |
Uptime: | 123:32:11 |
Calls: | 663 |
Calls today: | 11 |
Files: | 95,162 |
D/L today: |
1,680 files (212M bytes) |
Messages: | 299,178 |
Posted today: | 7 |