Zyada tar Clinical AI Validation Studies Gumrah Kun Kyun Hain

Jab koi clinical AI company validation results publish karti hai, to headline metric taqreeban hamesha AUROC hota hai: Area Under the Receiver Operating Characteristic curve. 0.95 ka AUROC bohat shandaar lagta hai, aur ye pitch decks, press releases aur conference talks mein is baat ke saboot ke tor par baar baar aata hai ke system "kaam karta hai." Hospital administrators, investors, aur kuch clinicians bhi 0.95 dekh kar ye nateeja nikal lete hain ke system 95% waqt target condition ko sahi pehchanta hai.

Ye matlab ghalat hai.

AUROC discrimination ko naapta hai: system ki ye salahiyat ke woh randomly chune gaye positive case ko randomly chune gaye negative case se upar rank kare. Ye tamam mumkina classification thresholds ka ek khulasa hai, jin mein se zyada tar amal mein kabhi use hi nahi honge. Baqi sab barabar ho to 0.95 AUROC wala system 0.80 wale se ranking mein behtar hai. Lekin AUROC ye kuch nahi batata ke us khaas operating threshold par kya hota hai jo hospital asal mein deploy karega, aur ye bhi nahi ke system us population mein kaisa chalta hai jise hospital asal mein serve karta hai.

Ek sepsis detection system jis ka AUROC 0.95 hai, sensitivity 85% aur specificity 90%. Paper mein ye numbers mazboot lagte hain. Ab inhein ek asal ICU mein rakhein jahan sepsis ki prevalence 5% hai.

Har 1,000 screen hone wale mareezon mein: 50 ko sepsis hai, 950 ko nahi. System 50 sepsis mareezon mein se 42.5 ko sahi pehchanta hai. Sepsis ke baghair 950 mein se 855 ko sahi clear karta hai. Lekin unhi 950 non-sepsis mareezon mein se 95 false alarms bhi deta hai. Positive predictive value, yani ye probability ke flag hone wale mareez ko waqai sepsis hai, 42.5 / (42.5 + 95) = 30.9% banti hai.

Is system ko use karne wala clinician har ek sahi alarm ke muqable mein taqreeban do false alarms dekhega. Masroof ICU mein jo system alert dete waqt 70% ghalat ho, woh chand hafton mein shor samjha jane lagta hai. 0.95 ka AUROC sach hai, lekin woh false positives ke us bojh ko chhupa deta hai jo tay karta hai ke koi is system ko asal mein use karega ya nahi.

Maine ye silsila clinical training ke dauran hi dekh liya tha, AI mein kaam karne se bohat pehle. EHR ke alerts se thakan itni shadeed thi ke physicians ke haath alert ka text parhe baghair "override" click karne ke aadi ho chuke the.

Zyada tar clinical AI validation studies system ko usi institution ke data par test karti hain jis ne usay banaya. Wajah amli hai: developers ke paas apna data hota hai, woh apne documentation ke patterns samajhte hain, aur internal validation tez aur sasti hoti hai. Lekin is se ek nizami bias paida hota hai jo report hone wali performance ko barha chadha deta hai.

Hospitals aise farq rakhte hain jo algorithm ki performance par gehra asar dalte hain. Documentation ke tareeqe alag hain: kuch institutions vital signs har ghante chart karte hain, kuch har chaar ghante baad. Labs order karne ke patterns alag hain. EHR configurations standard nahi. Jab model ek hi institution ke data par train aur test hota hai, to woh asal clinical signals ke saath saath un tamam site-specific cheezon ke patterns bhi seekh leta hai. Internal validation dono ko pakarti hai. External validation batati hai ke kaun se signals asli hain.

Sab se sabaq amoz misaal Epic ka sepsis prediction model hai. Internally train aur validate hone par is ne mazboot discrimination dikhai. Jab University of Michigan ke researchers ne usi model ko apne mareezon par azadana tor par evaluate kiya, to AUROC gir kar 0.63 par aa gaya, jo chance se bas thora behtar hai. Wong et al. ki ye study, jo 2021 mein JAMA Internal Medicine mein chhapi, is nateeje par pohanchi ke model ne sepsis ke do tihai cases miss kiye aur itne false alerts diye ke clinicians ka bharosa tezi se khatam ho gaya.

Korean Journal of Radiology mein ek systematic review ne diagnostic imaging AI ki 516 published studies ka jaiza liya aur paya ke sirf 6% ne apna algorithm kisi azad, external dataset par test kiya. Jo hospitals baqi 94% ki buniyad par khareedne ka faisla karte hain, woh ek haqeeqi maani mein aise performance claims khareed rahe hain jin ki tasdeeq nahi hui.

Koi company bilkul durust numbers report kar ke bhi gumrah kun tasveer bana sakti hai. Baat dhoke ki nahi. Baat framing ki hai.

Chuninda subgroups ki reporting: jis system ka overall AUROC 0.82 hai, woh 65 saal se zyada umar aur kam az kam teen comorbidities wale mareezon mein 0.91 tak pohanch sakta hai. Company "high-risk population" ke liye 0.91 ka figure report karti hai, jo technically durust hai, lekin ye tasur deta hai ke system poori mareez population mein jitna kaam karta hai, us se behtar hai.

PPV ke baghair sensitivity aur specificity report karna: jo company kehti hai "92% sensitivity aur 88% specificity," woh sachche numbers bata rahi hai jo shandaar lagte hain. Jo company kehti hai "4% prevalence par 28% PPV," woh usi system ki baat kar rahi hai. Sales presentation mein aap kaun sa number pehle rakhenge?

Zimmedarana validation report us threshold par performance dikhati hai jo deployment mein asal mein use hota hai, sirf tamam mumkina thresholds par nahi. Woh dikhati hai ke system mukhtalif qism ke hospitals aur mareez populations mein kaisa chalta hai, sirf ek pooled number ki shakal mein nahi. Aur woh batati hai ke validation internal thi ya external. Zyada tar reports aisa nahi kartin.

Clinical AI ka jaiza lene wale zyada tar health systems ko ek slide deck dikhaya jata hai jis mein performance ka ek hi number hota hai. Ye number aam tor par AUROC hota hai, ek aisa metric jise samajhne ki training zyada tar clinicians aur hospital administrators ko kabhi mili hi nahi. Ye naapta hai ke system mareezon ko risk ke hisaab se kitni achi tarah rank karta hai, ye nahi ke jab woh asal mein alert de to kitna durust hota hai. Zyada AUROC ka matlab ye nahi ke system aap ke hospital mein, aap ke mareezon ke saath, us threshold par jo aap use karenge, acha kaam karega.

Is se zyada ahem baatein ye hain. Kya system un hospitals mein test hua jo usay banane mein shamil nahi the? Agar sirf internally validate hua hai, to ho sakta hai performance aap ke institution mein qaim na rahe. Mukhtalif hospitals alag tarah document karte hain, labs alag frequency se order karte hain, aur alag populations serve karte hain. Jo system wahan kaam karta hai jahan bana, us ne is baare mein bohat kam sabit kiya hai ke woh aur kahan kaam karega.

Aap ki nurses har shift mein kitne false alarms dekhengi? Agar system har unit mein 20 alerts deta hai aur un mein se 15 ghalat hain, to aap ka staff chand hafton mein sab ko nazar andaz karna seekh lega. Us asal threshold par false positive rate maangein jo system use karta hai, validation paper ka nazariyati best case nahi.

Ye masle asal mein kitni jaldi pakarta hai? Jo system crisis se 2 ghante pehle deterioration pakar le, woh kuch faida deta hai. Jo ek din ya us se bhi pehle pakar le, woh aap ki team ko bilkul alag qism ki interventions ka waqt deta hai. Asal deployments ka median lead time maangein, retrospective analysis wala nahi.

Kya aap ke clinicians dekh sakte hain ke system ne kisi mareez ko flag kyun kiya? Agar jawab nahi hai, to accuracy jitni bhi ho, adoption kam rahega. Physicians ki training evidence parakhne ki hoti hai, be-wajah scores par amal karne ki nahi.

ML community ne discrimination metrics ke liye musalsal optimize kiya hai. AUROC aur AUPRC is sawal ka jawab dete hain: kya system positive aur negative cases ko alag kar sakta hai? Clinical deployment ke liye ye zaroori hai, lekin kaafi nahi.

Calibration naapti hai ke predicted probabilities apni asal qeemat mein bharosemand hain ya nahi. Achi tarah calibrated system agar kisi group ko sepsis ka 30% risk deta hai, to us group mein taqreeban 30% ko waqai sepsis hona chahiye. Kharab calibration ka matlab hai ke numbers par jaise hain waise bharosa nahi kiya ja sakta, aur jo clinician watchful waiting aur aggressive intervention ke darmiyan faisla kar raha hai, usay pata hona chahiye ke "70% risk" ka matlab waqai 70% hai ya nahi. Calibration ko calibration curves, Brier scores aur Hosmer-Lemeshow statistic se parkha ja sakta hai. Clinical AI ki publications mein ye shayad hi kabhi report hoti hai. Is ki ghair maujoodgi ko khatre ki nishani samajhna chahiye.

Lead time naapta hai ke system aam tareeqon ke muqable mein conditions ko kitna pehle pakarta hai. Jo systems "early warning" dene ka daawa karte hain, un ke liye asal sawal sirf ye nahi ke woh deterioration pehchan sakte hain ya nahi, balke ye ke standard clinical assessment ke muqable mein kitna pehle pehchante hain. Kam AUROC lekin 20 ghante zyada lead time wala system shayad us system se zyada jaanein bachaye jis ka AUROC zyada ho lekin jaldi pakarne ka faida na hone ke barabar.

Hum khud ko bhi unhi standards par parakhte hain. Hamari apni validation, calibration aur lead time samet, mukammal hone par publish ki jayegi.