From Devices to Data Models: The Future of AI for Surgeries

Technology has come a long way in developing and implementing autonomous functions in everyday life and in healthcare. Physicians are inundated with advanced devices and data-driven analytics that make promises of improving quality of care while making processes easier, more accurate, and more efficient.

AI models can potentially help enhance surgical precision, improve care, and predict and prevent adverse outcomes. But exploring the potential of AI in the operating room requires the prerequisites of strong datasets and adequate training, both for AI models and the surgeons themselves.

Robotic Surgical Tools as Data Collectors

One example is minimally invasive surgery. The majority of gallbladder removals are performed laparoscopically, with a robotic snipping tool and a camera introduced through small abdominal incisions. The surgeon can view the abdominal cavity on a video feed and guide the instrument’s movements, saving time and offering the patient a shorter recovery with minimal scarring.  

Just as these devices assist the physician performing surgery, they also work effectively as data collectors, gathering information that reveals surgical movements in real time and using the data to improve the surgical process. Laparoscopic video feeds and robotic instrument data give feedback on tool positioning, for example, helping surgeons understand where is a safe and unsafe place to dissect. We can segment out anatomy and, importantly, understand which step of the operation is happening when, just by looking at the video feed alone.

AI Tools That Improve Visual Perception in Surgery

There has been a surge in the tools and technology developed around improving visual perception in surgeries. These innovations can provide insight into the surgical scene as well as the patient experience and prognosis.

One urologist collected data from multiple prostatectomies to create a visual representation of a surgeon’s movements, or gestures. The model mapped each patient’s procedure, with different colors along a bar indicating either cold gestures (manual movements like cutting tissue with scissors) or hot gestures (electrical movements, such as cautery). The urologist was able to compare the surgical movements in procedures where patients had recovered erectile function within 12 months to those in surgeries where patients had not. The model could then be applied during procedures to guide surgeons on the gestures that were more likely to produce a positive outcome and help them avoid or adjust those that were not.      

Will Robots Perform Surgery?

With the growing use of large language models (LLMs) and visual language models (VLMs) that generate images, many people make assumptions about what these models can do.

For example, success was noted in robotic-assisted gallbladder surgeries performed on eight pig models. The robot autonomously and systematically clipped the organ at three different places. This machine, however, was not trained explicitly to do surgery. It watched surgeons perform the procedure and, through imitation learning, gathering and translating data, and feeding itself that data, the robot was able to perform the motions itself.

Measuring the performance of data models in a meaningful way is a challenge. Accuracy may be the objective in scientific data models, but applying this standard to the clinical setting adds a layer of complexity, especially when accuracy may mean missing something critical in patient evaluation and diagnosis.

Using a model that, for example, is 99 percent accurate in identifying the pixels or voxels (volumetric pixels) of a lesion could be beneficial in a search for cancer. However, that accuracy doesn’t mean much if it visualizes every bit of one lesion while missing another entirely, one that may critically impact a patient’s treatment course.

It’s also important to get data not just from large research hospitals but from smaller, community-based hospitals, where much of the care in the United States is delivered.  Professional societies and engineering groups have partnered to build data models that can be incorporated into various functions in surgery. The Society of American Gastrointestinal and Endoscopic Surgeons (SAGES), for example, partnered with the Medical Image Computing and Computer Assisted Intervention Society to collect 1,000 videos of laparoscopic cholecystectomies from around the world.

The expected questions when implementing data models meant to assist surgeons in the operating room include:

• Where and how was the model trained?
• Can the technology be trusted?
• Will this technology make me a better surgeon?
• Does the model think like me?

The last one may be particularly interesting when you consider that some models provide text explanations for their decisions (e.g., this is why this green area is a safe place for this incision, and why the red area is not). Some of that machine “thinking” may or may not align with how human beings think about it.

AI models should cover the checklist of reasoning that they’re built with. Their functions, including explanations, should represent or align closely with the clinical setting and with how a surgeon would experience a procedure, their thought processes, expectations, and how they might conceptualize problems.

What Could Go Wrong?

Part of the discussion around supplemental AI processes is anticipating mistakes in data model training, in the surgeon’s training, and in actual practice.

A lack of good datasets has made training AI models for tracking foreign objects difficult. Current models have limited video data—about 15 minutes—to work with, and surgeries can last anywhere from about 10 minutes to several hours.

I am working with partners to create better models using intuitive question-and-answer data. The questions would parallel what a surgeon might ask at different points during an operation: What time was this sponge inserted? Where was it last seen? Did I overlook any foreign objects?

The effectiveness of these data models will depend on the answers to key questions: Can the physician interpret output from one of these models and use it to optimize decision-making while operating? Does the interface obscure the view? And is the surgeon taught to recognize points where the AI might fail, where a direction could or should be ignored? What if the surgeon misses something that should not be ignored? And, do these systems leave room for the human experience or getting a second look by another physician?   

Co-Pilots in the Operating Room

SAGES has furthered efforts to reach a variety of physicians to learn how these AI technologies could be applied effectively to their practices. And while none of these have been put in practice yet, many systems are being evaluated by the FDA or used in training simulations with my residents.

Companies are submitting software models for real-time clinical decision support, including those that help in structure identification. And it shouldn’t be long, about 2 to 3 years, before we begin to see these things get clearance for use, essentially as decision-based “co-pilots” in the operating room. Fully autonomous systems for specific tasks could appear in a 5- to 10-year timeframe.

Doctors and the general public will ask the natural questions, respectively: When can a robot perform my surgery for me? Am I okay letting a robot operate on me? The answer is that it can already. It would take all day right now, but it’s not so much a technical barrier to autonomy. It’s really going to be more a question of implementation and a cultural acceptance by us as human beings.

Daniel Hashimoto, MD, MTR, spoke at the 2026 TDC Group Executive Advisory Board Meeting in Napa, California, which gathered top healthcare executives, academic researchers, and clinical leaders to discuss “Healthcare AI in 2026: Emerging Issues as the Field Matures.”

Dr. Hashimoto is an incoming Associate Professor of Surgery at Duke University and Director of Surgical Informatics for Duke University Health System. Most recently, he was Assistant Professor in the Department of Surgery at Perelman School of Medicine and in the Department of Computer and Information Science at the School of Engineering and Applied Science at the University of Pennsylvania, where he also served as director of the Penn Computer Assisted Surgery and Outcomes (PCASO) Laboratory.

Dr. Hashimoto earned his doctor of medicine (MD) degree and master of science (MS) degree in translational research at the University of Pennsylvania, completed his surgical residency at Massachusetts General Hospital (MGH), and completed a research fellowship in surgical artificial intelligence and innovation at MGH and the Massachusetts Institute of Technology. He completed a clinical fellowship in foregut surgery and comprehensive flexible endoscopy at University Hospitals Cleveland Medical Center. Dr. Hashimoto’s research has garnered awards from organizations such as the American Surgical Association, the Association for Surgical Education, the Society of American Gastrointestinal and Endoscopic Surgeons, and the American Society for Gastrointestinal Endoscopy. His work has been published in journals such as the New England Journal of Medicine, Nature Biotechnology, Nature Methods, and the Annals of Surgery. Dr. Hashimoto serves on the editorial boards of the Annals of Surgery, Computer-Assisted Surgery and the Journal Oversight Committee of Academic Medicine. He is Editor of the textbook Artificial Intelligence in Surgery: Understanding the Role of AI in Surgical Practice.

The opinions expressed here do not necessarily reflect the views of The Doctors Company. We provide a platform for diverse perspectives and healthcare information, and the opinions expressed are solely those of the author.