New York's AI "professional advice" bill would hurt the people who can't afford the alternative

The bill targets chatbots posing as licensed professionals. As written, it makes almost any useful answer in a regulated field a liability, with no defense for accuracy nor giving context.

Photo by Tingey Injury Law Firm on Unsplash

When my friend’s wife had a stroke, he used AI to check her new medications. Years before, she had gastric bypass surgery. He wanted to know if the surgery would change how the drugs worked.

It did. One of the main drugs is absorbed in the part of the intestine that her surgery had bypassed. He flagged it for her doctors. They adjusted the plan. AI may have saved her life.

My friend is a scientist. Given time, he could have found this himself. But in that emergency, there was no time. AI gave him a fast second opinion.

This kind of fast, informal help usually comes down to who you know. You get it if you can do the research yourself, or if you have someone to call.

Growing up working class, we felt lucky to know a doctor, a nurse, or a lawyer. They were the people you called when the system got confusing and you couldn’t afford to pay someone to explain it. Is my landlord required to give me hot water? Can I sue to get my deposit back? If you knew the right person, you got a straight answer.

In high school, I turned down AP courses because I couldn’t pay the test fees. No one told me waivers existed. Years later, I stumbled onto a list of colleges that cover full tuition for low-income students. It changed my life. A college counselor might have handed me that list if I’d had one.

AI gives that same kind of information to people who don’t know the right person. Even though AI makes mistakes and isn’t fully reliable, it still helps people do better research in their language while they wait weeks for an appointment. It can help someone put their symptoms in context instead of panicking. These benefits don’t show up in GDP or in headlines.

That is why a new bill in Albany worries me.

A well-meaning bill

New York Senate Bill 7263 tries to stop AI from posing as a licensed professional, which makes sense to me. If a chatbot pretends to be your doctor and hands you bad advice, you should be able to sue.

But the bill is written so broadly that it would punish almost any AI that gives useful information regarding a “licensed field”.

The bill makes anyone who "owns, operates, or deploys" a chatbot liable if the bot gives a "substantive" answer regarding medicine, law, nursing, veterinary care, pharmacy, social work, and mental health counseling, among others. A disclaimer like "I am not a doctor" does not protect the chatbot’s deployer.¹

The biggest problem is that "substantive response" is not clearly defined. ChatGPT gave my friend information about how his wife's medications would interact. Is that substantive? Does substantive mean “could be helpful”?

Because the term is so vague, an AI company or deployer can only protect itself by refusing to give useful answers. The bill offers no other path. Lawyers call this strict liability. You pay when something goes wrong, no matter how careful you were.

It does not matter whether the answer was accurate, or whether the bot listed every risk and warned that it might be wrong. The liability is the same. A New York law firm warns that operators will simply shut off whole categories of answers, since no disclaimer removes the risk. The only safe move is to say nothing that looks substantive.

Who this hurts

We should want AI to give people context and useful information. Many people have no steady access to a lawyer or a doctor.

In New York in 2010, 98 percent of tenants in eviction cases and 95 percent of parents in child support cases went to court with no lawyer. For many of them, the choice was not AI versus a professional. It was AI versus a Google search, or nothing.

I want AI to help a tenant who can’t afford a lawyer understand a confusing eviction notice.

The bill also turns lawsuits into a business. If a violation is “willful,” the operator has to pay the plaintiff’s legal fees. Shifting the fee can be a good thing, but it also rewards lawyers for filing lots of suits and winning only a few. Even when the harm is small or unclear, most AI companies or deployers will settle, because settling is cheaper than going to court. A lawyer could build a bot whose only job is to quiz every chatbot in the state and flag any “substantive” reply.

Big companies can probably absorb this. I worry about the smaller groups. A small nonprofit that wants to deploy a chatbot trained on local domestic violence resources so a survivor can get answers at 2 a.m. while she waits for a caseworker. Or someone with long COVID who builds a custom GPT trained on the latest research and shares it for free. Or a nonprofit law firm that wants to test a bot trained on visa options for Spanish-speaking immigrants.

None of these makes money. These deployers would be taking on legal risk they can’t afford.

What a better bill would do

I want people to be free to build better tools for each other. A good bill would target only operators who claim to be licensed or fake a credential. If it goes further into “substantive” answers, it should at least protect information that is accurate or accompanied by context, rather than letting anyone sue any operator who tries to help.

As AI gets better, I expect some of the lawsuits to come from businesses that don't want the competition. A licensed professional could use this law to go after a tool that was starting to take their customers.

This can be fixed. California’s version, AB 489, is narrower. It also lets state agencies enforce the rule instead of opening the door to a lawsuit from anyone.

It’s unpopular to talk about AI’s upside, but this is part of the problem. The harms make the news, but the benefits are hard to count. Thus, we write laws from a lopsided picture. The costs won’t be clear either. We won’t see the careful, useful tools a nonprofit or a volunteer would have built.

We want more places giving careful, useful information, not fewer. For lower-income people, the status quo is already no information. A law that leaves them with even less is not consumer protection.

Note: I didn’t dig into whether this bill affects Anthropic or OpenAI directly. One law firm says it doesn’t; other sources say it does. The law may apply when you use their models directly, but not when someone else deploys their model inside another product. I stuck to the parts I’m sure about.

Fact checked with Tom Walczak’s Claude skill.

1. Even though lawmaker Kristen Gonzalez states the bill only targets AI systems that claim to be medical professionals, Section 1C says: “(c) An operator shall not waive or disclaim liability under this section by notifying consumers that they are interacting with a non-human chatbot system.”