"Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not."
It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.
Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.
A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.
A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.
Even the premise seems to not understand what it's trying to require. There are zero humans accessing a website by sending electrical signals to Comcast using a telegraph or pulsing a flashlight into a strand of fiber. Every web browser is a computer program running on a computer.
So is a web browser a bot? What about an ad blocker? RECAP? NewPipe? The default browser opening a link at the behest of some other program? A browser extension that downloads all your emails from your webmail provider?
Their intention is presumably not to ban those things, or equivalently, legally require them to declare themselves to be something that would cause them to be blocked. Or if it is then they're proposing something hostile and malicious. But if that isn't their intention then where is their method of distinguishing that from whatever it is they're actually trying to do?
A web browser isn't a bot as long as it's controlled by a human but it becomes one once it's automated.
You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
Any bot is operating on behalf of one or more humans somewhere. Bots are good, for the most part.
The conventional way of doing web isn't efficient. If it were, no one would be complaining about bots. The way we do web is the thing that needs to be updated.
What gets me is how much the people who advocate for the "open web" have completely incorporated the values of Facebook and Google.
Like there's that anime girl who harasses you when you visit so many open source web sites that just continues the pattern of being harassed by cookie dialogs and consent dialogs and pop-up ads and being asked for your email address and of course there is Wikipedia which is doing more and more of the same.
Another web is possible but web populists don't really believe it.
Just to get started, bots have been better at filling out CAPCHTAs than humans for a long time. I knew someone who got his site hacked by Russians circa 2008 because he published a neural network in freakin PHP that could read them. Anubis causes humans stress, slows down task completion, probably raises peoples blood pressure, and cumulatively causes a great deal of harm to users the same way Cloudflare's system does. Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.
CAPTCHAs, cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits with triple damages.
What's the alternative? Invisible algorithmic fuzzy recognition on the server side that just randomly bans you if you look like a bad bot, with no recourse? At least if you solve a captcha and dismiss the cookie banner you can access the website.
> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.
Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.
> Another web is possible but web populists don't really believe it.
How? Sharing knowledge with the world used to be an obviously good thing. Now it's a way to help tech companies replace people with machines. What alternative web could change this state of things?
>"In 2022, a Pulitzer-winning Wall Street Journal investigation analyzed over 31,000 financial disclosures from 12,000 senior government employees, revealing that nearly a quarter of federal officials traded stock in industries their agencies regulated."
Ya know, on the one hand, I wonder how many federal officials own nVidia stock and stock of other AI companies...
But then again, on the other, I wonder how many nVidia employees (and those of other AI companies!) own government T-bills (and other government backed securities!) in their 401K or other investment accounts!
You know, everybody in large corporations and governments owning shares in other large corporations and governments -- adds a whole new level of meaning to "stakeholder capitalism"... :-)
(In the far future, there'll probably be small foreign governments which become their own ticker symbol -- and get publicly traded on various stock exchanges! :-) You know, like: "Would you like to buy a share in Denmark (DNMK) or perhaps Norway (NRWY)? I hear they're very competitive with Apple (AAPL) right now! Great price toearnings (PE) ratio!" :-) )
You can already buy shares in a nations economy - it is called owning currency. The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).
I am simplifying terribly but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
>You can already buy shares in a nations economy - it is called owning currency.
An interesting point! Yes, that is definitely one way that currency could be looked at...
>The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).
Agreed -- but if we were really to analyze things, scrutinize the way things really work at the highest degree of resolution, then we would have to observe that:
a) There are multiple countries/currencies in the world.
b) That each country's/currency's inflation rate is different than all of the other countries/currencies.
c) If each country's/currency's inflation rate is different, this means that each countries's issuance of new shares (and control of interest rates) -- their financial policies, are, by comparison, more effective or less effective than those of other countries...
Phrased another way:
All countries do not control the rate of inflation of their currencies equally.
Phrased another way:
Some nations are better at controlling their currency's rate of inflation than other nations.
True or not true?
>I am simplifying terribly
As am I! :-)
>but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
> In 2022, Tech Policy Press received a grant from the Omidyar Network for a research project evaluating vulnerabilities in the design, engineering, and use of encrypted messaging applications.
https://omidyar.com/ is geofenced for me, presumably because they identified me as being anti open web.
The article is ai lobby intended to repel a bill that would
> allow the FTC to pursue penalties of up to $53,000 if a person uses a stealth bot in a way that is “reasonably likely to damage, impair, or burden” a website’s “technical or commercial operation.”
The definition of "stealth bot" is a bot that does not identify itself as such and their main grief to the proposal is
> If researchers cannot easily understand what it means to “deploy” a crawler in a prohibited way, or what activity is “reasonably likely to burden or impair a website,” they may refrain from scraping rather than risk violating the law.
Which is the biggest load of shit I've heard given that all you need to do is identify yourself as a bot and respect robots.txt or be banned, the status quo before a bunch of assholes started systematically destroying the internet.
You can still ban everyone who doesn’t respect robots.txt. My experience with a crawler is that people don’t actually provide a robots.txt far more often. I can back off on 429 but I’d rather use Crawl-Delay etc.
Bots have limited ability to announce themselves. I just use fixed IP range and UA. But the Chinese trawlerbots use smart TV residential proxies and will copy someone else’s UA.
Those would fall under the "stealth" bots subject to a fine.
The whole point is that if you can prove a company is doing this kind of shit, they should be on the hook for it. Of course, I'm not expecting any company in China to actually pay the fines, but this opens up some options such as seizing foreign assets.
It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.
A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.
A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.
So is a web browser a bot? What about an ad blocker? RECAP? NewPipe? The default browser opening a link at the behest of some other program? A browser extension that downloads all your emails from your webmail provider?
Their intention is presumably not to ban those things, or equivalently, legally require them to declare themselves to be something that would cause them to be blocked. Or if it is then they're proposing something hostile and malicious. But if that isn't their intention then where is their method of distinguishing that from whatever it is they're actually trying to do?
You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
The conventional way of doing web isn't efficient. If it were, no one would be complaining about bots. The way we do web is the thing that needs to be updated.
Like there's that anime girl who harasses you when you visit so many open source web sites that just continues the pattern of being harassed by cookie dialogs and consent dialogs and pop-up ads and being asked for your email address and of course there is Wikipedia which is doing more and more of the same.
Another web is possible but web populists don't really believe it.
Just to get started, bots have been better at filling out CAPCHTAs than humans for a long time. I knew someone who got his site hacked by Russians circa 2008 because he published a neural network in freakin PHP that could read them. Anubis causes humans stress, slows down task completion, probably raises peoples blood pressure, and cumulatively causes a great deal of harm to users the same way Cloudflare's system does. Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.
CAPTCHAs, cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits with triple damages.
Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.
> Another web is possible but web populists don't really believe it.
How? Sharing knowledge with the world used to be an obviously good thing. Now it's a way to help tech companies replace people with machines. What alternative web could change this state of things?
Ya know, on the one hand, I wonder how many federal officials own nVidia stock and stock of other AI companies...
But then again, on the other, I wonder how many nVidia employees (and those of other AI companies!) own government T-bills (and other government backed securities!) in their 401K or other investment accounts!
You know, everybody in large corporations and governments owning shares in other large corporations and governments -- adds a whole new level of meaning to "stakeholder capitalism"... :-)
(In the far future, there'll probably be small foreign governments which become their own ticker symbol -- and get publicly traded on various stock exchanges! :-) You know, like: "Would you like to buy a share in Denmark (DNMK) or perhaps Norway (NRWY)? I hear they're very competitive with Apple (AAPL) right now! Great price toearnings (PE) ratio!" :-) )
I am simplifying terribly but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
An interesting point! Yes, that is definitely one way that currency could be looked at...
>The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).
Agreed -- but if we were really to analyze things, scrutinize the way things really work at the highest degree of resolution, then we would have to observe that:
a) There are multiple countries/currencies in the world.
b) That each country's/currency's inflation rate is different than all of the other countries/currencies.
c) If each country's/currency's inflation rate is different, this means that each countries's issuance of new shares (and control of interest rates) -- their financial policies, are, by comparison, more effective or less effective than those of other countries...
Phrased another way:
All countries do not control the rate of inflation of their currencies equally.
Phrased another way:
Some nations are better at controlling their currency's rate of inflation than other nations.
True or not true?
>I am simplifying terribly
As am I! :-)
>but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
Again, an interesting point!
And yes, I could agree with that!
https://omidyar.com/ is geofenced for me, presumably because they identified me as being anti open web.
The article is ai lobby intended to repel a bill that would
> allow the FTC to pursue penalties of up to $53,000 if a person uses a stealth bot in a way that is “reasonably likely to damage, impair, or burden” a website’s “technical or commercial operation.”
The definition of "stealth bot" is a bot that does not identify itself as such and their main grief to the proposal is
> If researchers cannot easily understand what it means to “deploy” a crawler in a prohibited way, or what activity is “reasonably likely to burden or impair a website,” they may refrain from scraping rather than risk violating the law.
Which is the biggest load of shit I've heard given that all you need to do is identify yourself as a bot and respect robots.txt or be banned, the status quo before a bunch of assholes started systematically destroying the internet.
Bots have limited ability to announce themselves. I just use fixed IP range and UA. But the Chinese trawlerbots use smart TV residential proxies and will copy someone else’s UA.
I think many don't see the point of robots.txt as it will just be ignored by most bots anyway, so instead concentrate on more complicated measures.
The whole point is that if you can prove a company is doing this kind of shit, they should be on the hook for it. Of course, I'm not expecting any company in China to actually pay the fines, but this opens up some options such as seizing foreign assets.