The synergistic effect of fracture development and high-temperature conditions in deep coal seams significantly exacerbates drilling fluid performance degradation and loss circulation risks. Achieving coordinated optimization of enhanced sealing and reservoir protection remains a critical technical challenge in current drilling fluid technology. To address this issue, this study systematically conducted high-temperature high-pressure (HTHP) filtration, sealing performance, and core permeability damage experiments. Three high-temperature resistant additives—modified ketal (NAX50), lignite resin (SPNH), and graft copolymer (JZW)—were screened, and their optimal dosage ranges were determined. The Analytic Hierarchy Process (AHP) was employed to establish a Plugging-Protection Balance Index (PPDI), which was combined with Box-Behnken response surface methodology to optimize the composite system, thereby revealing the influence mechanisms of key factors and their interactions on the PPDI. Based on the characteristics of the 3D response surface and scatter distribution of PPDI, two types of composite drilling fluid systems were developed: one emphasizing low damage and the other prioritizing high plugging efficiency.Optimization results indicated that the optimal formulation was 2.6% NAX50 + 5.054% SPNH + 2.205% JZW, with NAX50 and JZW exhibiting significant synergistic effects. Comprehensive performance evaluation demonstrated that the Type II system exhibited excellent rheological properties, inhibition, and plugging stability, with a reservoir damage rate of only 10.65%. In contrast, the Type III system achieved a 22% reduction in fluid loss and a 6.93% improvement in plugging efficiency, but at the cost of a 16.37% increase in damage rate. Both systems were non-toxic and demonstrated good biodegradability, meeting environmental requirements for drilling operations. The multi-objective optimization method and composite drilling fluid system proposed in this study provide a theoretical foundation and engineering value for the optimized design of drilling fluids in deep coal seam applications.