Why a Karnataka High Court Judge Warned Against Using AI in Courtrooms

Justice M. Nagaprasanna of the Karnataka High Court recently raised serious concerns about the use of artificial intelligence (AI) in the legal field, especially in judicial decision-making. His warning came during a high-profile case involving X Corp (formerly Twitter) and the Indian government's ‘Sahyog’ portal — a platform used to issue content-blocking orders.

“AI Is a Serious Threat to the Legal Profession”:

While hearing the case, Justice Nagaprasanna didn’t hold back his thoughts on AI in the courtroom. He strongly cautioned against using AI tools to write or draft court judgments.

“The most dangerous thing is using AI to draft or write judgments,” he said.

He added that relying too much on AI could eventually “destroy the legal profession”, urging lawyers and judges to stay cautious.

“Dependence on artificial intelligence should not make your intelligence artificial.”

Fake Judgments and Citations: The Risks of AI:

Solicitor General Tushar Mehta, appearing for the Central government, also shared examples of how AI can go wrong in legal settings.

He spoke about real cases where lawyers submitted AI-generated content in court—some of it entirely fabricated. In one situation, a lawyer presented a judgment that AI had made up, claiming it was in his client’s favor. In another, fake citations were used to try and win litigation costs.

Mehta pointed out that AI has a tendency to "hallucinate", meaning it can produce believable but false information — a huge risk in legal matters.

The Bigger Picture: X Corp vs The Government:

The larger case being heard by the court is also about technology — but in a different way. X Corp has challenged the Indian government’s ‘Sahyog’ portal, arguing that it allows content takedowns without following proper legal procedures under Section 69A of the Information Technology Act, 2000.

This section of the law was previously upheld in the Supreme Court’s Shreya Singhal judgment, which set clear safeguards for how and when content can be blocked online. X claims that Sahyog bypasses these safeguards and gives government agencies too much unchecked power.

The company also says the government is misusing Section 79(3)(b) — a rule meant to define the limits of intermediary liability — to run a “parallel and illegal” takedown system.

This complaint came after several posts were ordered to be taken down by the Ministry of Railways, which reportedly used the Sahyog portal to block content related to a stampede at New Delhi Railway Station.

Balancing Free Speech and Public Safety:

During the hearing, the court also discussed larger concerns — like clickbait content, misinformation, and how to balance free speech with public interest in the digital age.

SG Mehta argued that companies like X are not just passive platforms but profit-driven businesses that amplify content through algorithms. He said that notifying such platforms to remove unlawful content does not automatically violate the right to free speech.

He also said the current legal and digital environment is very different from what existed during the Shreya Singhal case, suggesting that new rules may be necessary.

What Happens Next?

The case is ongoing and will continue on Friday. X Corp is asking the Karnataka High Court to:

• Declare that Section 79(3)(b) does not give the government power to issue such takedown orders,

• Quash the Sahyog-based orders, and

• Protect the platform from any coercive action until the case is decided.

  
  

Final Thoughts:

The case raises important questions about the role of AI in the legal profession and the future of online free speech in India. While AI can be a useful tool for legal research, experts warn that over-reliance on it can lead to serious mistakes — even fabricated court decisions.

As technology continues to evolve, courts, lawyers, and lawmakers will need to find the right balance between innovation, accountability, and human judgment.