This study presents a diagnostic framework for healthcare using explainable artificial intelligence and machine learning. Previous studies show that the ability of prediction models greatly depends on the relevance and independence of feature set and hence various feature selection methods are presented in the literature. An incremental feature selection is performed in this study to first select non-redundant features and then to select relevant features suitable for the problem domain. Further, the feature importance is set as the initial weights for the interpretable neural network and the results are presented. Further, an app for glioma grading is developed based on this approach and its predictive power is also presented.