Cauliflower (Brassica oleracea var. botrytis) is one of the most popular crops that are subject to a variety of diseases affecting the leaf apparatus, which impact quality and production. Despite the progress in deep learning, appropriate disease detection under real-field conditions remains a serious problem. This paper introduces an expert system GNN-PDP, which is a novel Graph Neural Network based model for the automated classification of cauliflower leaf diseases using images taken with a smartphone. A Region Growing Segmentation (RGS) is used to extract perceptual regions in this structure and statistical features are utilized as graph node features. The Salp Swarm Algorithm (SSA) finds optimal features that result in better generalization. A total of 750 images were gathered in four categories of diseases. The assessment was made based on accuracy, precision, sensitivity, specificity, and F1-score, in relation to Linear Discriminant Analysis (LDA), Random Forest (RF), Deep Neural Networks (DNN), and CNN classifiers. GNN-PDP achieved a superior classification accuracy of 89.0%, outperforming all other experiments. The model has great potential for smart agriculture in disease management.