High-Dimensional Regression Analysis and Artificial Intelligence: Theory, Methods and Applications

In this book, a novel high-dimensional linear and nonlinear regression model is introduced to address, in part, the challenges of evaluating the stability and confidence of large-scale models' interpretability. The book begins by reviewing foundational concepts in regression analysis, and discussing the current state and challenges of AI interpretability. Through an in-depth exploration of regression models, the core principles of data-driven linear regression are explained. To enhance the explanatory power of regression models, variable-parameter regression models are further investigated and extended to variable-parameter nonlinear regression models. To handle complex relationships, the Gauss-Newton iterative method is incorporated, ensuring the stability of high-dimensional nonlinear regression. The Confidence Interval-based Credibility Evaluation (CICE) framework combines statistical indicators—such as interval width, center deviation, and accuracy—into a single score to assess the stability and reliability of explanations, validated through case studies in engineering, finance, and time series prediction. Overall, the book presents a coherent framework for interpretable AI, integrating regression modeling, confidence region construction, and credibility evaluation to enhance interpretability and statistical accountability, fostering more trustworthy AI systems. Chapter 1 introduces the fundamental concepts and theoretical developments of both regression analysis and AI explainability, highlighting their interconnections. Chapter 2 reviews essential probability theory and mathematical statistics, covering random variables, measure spaces, probability distributions, parameter estimation (including least squares and maximum likelihood methods), and asymptotic theory, which serve as the foundation for analyzing model consistency and convergence. Chapter 3 focuses on the effects of correlated errors in linear regression, establishing parameter convergence conditions to ensure the consistency and asymptotic normality of covariance estimators. Chapter 4 introduces variable-parameter regression models and systematically studies M-estimators and generalized regression models within the framework of robust statistics. By addressing non-normal errors and outliers, these methods improve model adaptability. The chapter also establishes the robustness of the generalized regression model through theoretical analysis of covariance estimation. Chapter 5 introduces the Confidence Interval-based Credibility Evaluation (CICE) framework, which integrates multiple statistical indicators into a unified score to assess the stability and reliability of model explanations. Through real-world case studies in engineering, finance, and time series prediction, the effectiveness of CICE in detecting unstable interpretations and enhancing model transparency is demonstrated.