A chatbot powered by artificial intelligence, developed by New York City to assist small business owners, is facing criticism for providing peculiar advice that misinterprets local regulations and encourages businesses to break the law. Despite these issues being brought to light by the tech news outlet The Markup last week, the city has chosen to keep the tool available on its official government website. Mayor Eric Adams defended this decision while acknowledging that the chatbot’s responses were inaccurate in certain areas.
Launched in October as a resource for business owners, the chatbot offers algorithmically generated responses to queries about navigating the city’s bureaucratic processes. It includes a disclaimer stating that it may occasionally provide incorrect, harmful, or biased information, and has a strengthened warning that its responses do not constitute legal advice. However, it continues to offer incorrect guidance, raising concerns among experts about the risks associated with governments adopting AI-powered systems without adequate safeguards.
Julia Stoyanovich, a computer science professor and director of the Center for Responsible AI at New York University, criticized the rollout of the software without proper oversight, stating that it shows a lack of responsibility on the part of the authorities.
Responding to queries, the chatbot incorrectly stated that it is permissible for employers to terminate employees who report sexual harassment, fail to disclose a pregnancy, or refuse to cut their dreadlocks. It also contradicted the city’s waste initiatives by claiming that businesses can dispose of their trash in black garbage bags and are not obligated to compost.
At times, the chatbot’s responses bordered on the absurd. For instance, when asked if a restaurant could serve cheese nibbled on by a rodent, it replied affirmatively before adding that the extent of the damage caused by the rodent should be assessed and customers should be informed about the situation.
A spokesperson for Microsoft, which provides the bot through its Azure AI services, stated that the company is collaborating with city officials to enhance the service and ensure that its outputs are accurate and based on official city documentation.
During a press conference, Mayor Eric Adams defended the technology, suggesting that identifying issues is a normal part of refining new technology. However, Stoyanovich criticized this approach as reckless and irresponsible.
Experts have long raised concerns about the limitations of large language models like ChatGPT, which are prone to producing inaccurate and nonsensical answers due to being trained on vast amounts of internet text. Despite this, the public’s interest in such chatbots has led private companies to introduce their own versions, albeit with mixed results.
Jevin West, a professor at the University of Washington, highlighted the heightened stakes when such models are endorsed by the public sector, emphasizing the potential damage that incorrect advice could cause.
Other cities that utilize chatbots have typically restricted them to a narrower range of inputs to reduce misinformation. Ted Ross, the chief information officer in Los Angeles, explained that the city carefully curates the content used by its chatbots, which do not rely on large language models.
The issues with New York’s chatbot serve as a warning for other cities, according to Suresh Venkatasubramanian, the director of the Center for Technological Responsibility, Reimagination, and Redesign at Brown University. He emphasized the importance of cities considering the purpose of using chatbots and ensuring they address specific problems rather than merely replacing human interaction without gaining any benefits.
