Examination of an Image Identification System for Agricultural Machines Based on Machine Vision

Anzhi Chen1
1Sichuan Jiuruifeng Intelligent Technology Co., Ltd, Chengdu 610041, Sichuan, China

Abstract

As computer and mechanical automation technologies advance, machine vision-based non-destructive testing technology finds use in a multitude of domains. Non-destructive testing technologies can be used on apple sorting equipment to decrease apple damage while simultaneously increasing sorting efficiency. As a result, the apple sorting machine’s image identification system now incorporates machine vision technology. The automatic classification of apple grades is accomplished by gathering, processing, extracting, and computing the contour features of apple photographs using preset sorting levels. The automatic control system then sorts apples of different grades to designated locations, thus achieving the automation of apple sorting. Tests were run on the sorting machine’s image recognition system to confirm the solution’s viability. The outcomes demonstrate that the sorting machine can effectively classify fruit automatically based on their perimeter, which is important for fruit sorting automation.

Keywords: Apple sorting machine, Machine vision, Image recognition, Non destructive testing, Feature extraction

1. Introduction

The field of agricultural automation and computer vision has advanced significantly in recent years, especially in the area of non-destructive testing methods [1]. One prominent application of this progress is observed in the development of automated fruit sorting systems, which not only enhance efficiency but also mitigate potential damage to the produce. Apple grading detection is a crucial step in the commercial processing of apples, having an immediate effect on apple packing, shipping, storage, and income [2]. Traditional methods involve weighing apples and then grading them, using manual and mechanical devices, resulting in low sorting efficiency and potential damage to apples. Moreover, these approaches cannot provide a realistic evaluation of the colour and surface flaws of apples. Machine vision-based non-destructive testing technology has been used in a number of industries recently. Apple sorting devices that use machine vision into their design can successfully increase apple sorting efficiency and accomplish non-destructive apple testing [3].

The pivotal process of grading apples holds immense importance in the commercial handling of the fruit, directly impacting packaging, transportation, storage, and sales revenue. Traditionally, methods involved weighing apples and subsequently categorizing them. However, the utilization of manual labor and mechanical devices in these processes not only results in low sorting efficiency but also poses a risk of damage to the apples [4]. Furthermore, these methods fail to provide a comprehensive assessment of color and surface defects, limiting their effectiveness.

The integration of machine vision-based non-destructive testing techniques presents a transformative opportunity to address these challenges. By incorporating machine vision into the design of apple sorting machines, it becomes possible to significantly improve sorting efficiency and achieve non-destructive testing of apples. This marks a departure from traditional methods and opens avenues for a more comprehensive and nuanced evaluation of apples based on color, shape, and surface characteristics [5, 6, 7].

Central to the success of the image recognition system is the application of non-destructive testing techniques. These techniques play a critical role in the detailed analysis of apple images, enabling the system to identify color variations and surface defects accurately. By avoiding physical contact and damage to the apples during the sorting process, non-destructive testing ensures that the produce retains its quality and market value. The significance of non-destructive testing extends beyond sorting efficiency to the preservation of apple quality. The ability to identify and categorize apples without causing harm not only enhances the overall efficiency of the sorting process but also contributes to reducing post-sorting losses and maintaining the integrity of the fruit [8]. While the integration of machine vision and non-destructive testing techniques presents a promising avenue for automating apple sorting, several challenges persist in the current research landscape.

Apples exhibit a wide range of characteristics, including varying colors, shapes, and surface features. The complexity of these characteristics poses a challenge in developing a universal sorting criteria that accommodates the diversity observed in different apple varieties [9]. The efficiency of automated sorting systems is contingent upon real-time image processing capabilities. Achieving rapid and accurate image analysis, especially in high-throughput scenarios, remains a formidable challenge that requires continuous research and technological advancements [10]. To enhance the accuracy of sorting, integrating data from multiple sensors, such as color sensors and surface defect detectors, is essential. However, the seamless integration and synchronization of data from diverse sources present technical challenges that demand innovative solutions.

Addressing these challenges requires a concerted research effort, involving advancements in machine learning algorithms, sensor technologies, and system integration methodologies [11]. The ultimate goal is to develop a robust and versatile automated sorting system that can revolutionize the fruit industry by enhancing efficiency, reducing losses, and ensuring the delivery of high-quality produce to consumers.

This paper delves into the integration of machine vision technology into the design of an apple sorting machine, with a focus on image recognition systems. The objective is to explore the feasibility and efficacy of employing non-destructive testing techniques to not only elevate the sorting efficiency of apples but also to evaluate them without causing harm.

2. Machine Vision-based Agricultural Image Recognition System

The agricultural machinery image recognition system constitutes the core of machine vision. Visual recognition typically involves the initial acquisition of corresponding images by visual sensors. Subsequently, these images undergo processing by a machine vision system. After completing processes like feature extraction and picture segmentation, the system determines the image’s final content [9]. The process flow of agricultural image recognition systems advances from basic image collecting and preprocessing to intermediate processing layers that are capable of segmenting images and extracting features. Finally, it culminates in high-level processing that utilizes various algorithms to achieve plane recognition of the image.

2.1. Image Pre-processing

The collection and acquisition of images rely on visual sensors. After obtaining the images, preprocessing steps are necessary, including converting RGB values to HIS mode, enhancing defect features, and optimizing image resolution. Image preprocessing primarily involves four aspects. Firstly, grayscale processing transforms RGB tricolors, converting the acquired color image into a grayscale image, and reducing the 24-bit image to an 8-bit image [12]. Secondly, uniformization processing is applied to standardize the size of crop images, allowing for more characteristic points for subsequent recognition through uniform scaling. Subsequently, image enhancement is performed to improve clarity and strengthen recognition effects. Techniques such as dilation and erosion can be used to enhance image resolution or filter out interference in the image. Finally, for convenient image processing, binarization can be applied using the Niblack algorithm, determining the central coordinates on crop images and achieving binarization through calculating pixel point variances and means [11, 13].

2.2. Segmenting Images

To divide an image into several important pieces is the aim of image segmentation. Three primary segmentation techniques-based on image edges, image thresholds, and image regions-are available. Using edge detection algorithms, one of the segmentation techniques based on picture edges is to compute the grey value, colour, and texture of the image region with irregular edges, then utilise these edge contours to identify the image region [14].

Threshold-based image segmentation relies on determining a threshold based on the emission or absorption of light on the object’s surface. There are two main approaches to threshold-based segmentation. The first is manual segmentation, a fundamental method that uses histogram techniques to identify the optimal threshold and isolate the segmented target from the background. Manual segmentation is influenced by subjective factors, requiring repeated monitoring and adjustments to achieve satisfactory segmentation results. In agricultural production, crops are significantly affected by lighting conditions, necessitating threshold adjustments when changes occur [15, 16]. Frequent adjustments, however, may impact the effectiveness of threshold segmentation.

The second approach is automatic threshold segmentation, which eliminates the need for human intervention. In automatic thresholding, the threshold is automatically incorporated into the histogram information during the target segmentation process. Based on pre-set parameters, the segmentation operation is performed. However, threshold segmentation encounters challenges when dealing with images lacking distinct peaks and valleys, leading to difficulties in calculating effective thresholds and resulting in incomplete region segmentation.

Region-based segmentation is a method of segmentation based on combining representative pixel content within a region, resulting in a segmentation approach that performs less effectively in noise identification. This method is mostly applied to identify crops with distinct characteristics on the crop surface, focusing on recognition in areas such as the overall crop, roots, and defects. The segmentation results are illustrated in Figure 1.

Figure 1. Color Detection Results

3. Image Recognition System for Agricultural Machines

One fundamental skill in art and painting is the ability to interpret images, and correctly recognizing and interpreting paintings allows one to grasp the intention and deeper meaning of the artwork. The visual tendencies in contemporary art are closely related to the rapid development of visual culture. Currently, museums house a vast number of paintings, and different aesthetic approaches yield varying visual communication effects. The study of visual symbols in artworks is a topic worthy of in-depth exploration. If the methods of semiotics were introduced into the study of art and painting, effective identification methods for artworks could be developed, ultimately enhancing the ability to interpret artistic works [17].

The efficiency of sorting apples would be greatly increased by creating an apple sorting machine using machine vision based on the interpretation of paintings and other visual media [18]. This approach, utilizing non-destructive testing, can effectively reduce damage to apples during the sorting process. Apples that are sorted according to size will work better for processing, processing, and storage. Apart from that, apples with a form grade will be more marketable. Figure 2 shows the steps involved in sorting apples using machine vision technology.

Figure 2. Apple Sorting Device: A Non-destructive Sorting Method

Taking pictures of the apples is the initial stage in the apple sorting machine. Subsequently, machine vision systems are employed to process these images. Finally, based on the grading standards, the apples’ grades are determined. Once the apple grades are established, an automatic control system is utilized to sort apples of different grades into designated devices, thus achieving non-destructive testing and automated classification of apples [19, 20].

4. Apple Visual System Design for Sortingl

Manual sorting would lead to significant labour waste and lengthier sorting cycles in the apple sorting process. Mechanical sorting, while efficient, poses a risk of damaging the fruit [21]. Non-destructive testing, however, offers the advantage of improving sorting efficiency without causing harm to the fruit. During non-destructive testing of fruit quality, it is essential to assess factors such as apple size, surface defects, color, and shape, which serve as the basis for apple grading. Figure 3 depicts the sorting machine system’s schematic diagram.

Figure 3. Apple Non-destructive Sorting Machine Schematic Diagram

During non-destructive sorting, the primary process involves capturing intact images of apples. Subsequently, image processing, feature extraction, and contour analysis are employed to accomplish quality grading. The assessment of grades is based on factors such as apple size, color, and surface defects. The primary method used in apple size grading to calculate the diameter and circumference of the apple is to extract the shape contour from the image. Apple color grading involves converting RGB values to HIS mode, obtaining the hue histogram, and evaluating color characteristics. Apple surface flaws are evaluated by extracting visual characteristics and applying grayscale or image model techniques. As shown in Figure 4, there are six primary processes in the post-image acquisition processing: image grayscale processing, image normalisation, image enhancement, image binarization, and image feature extraction [22, 23].

Figure 4. Apple Image Processing Flow

The image’s basic properties are revealed by image processing, and these can be used to determine the apple image’s quality based on its size and surface characteristics. The following are the steps involved in the image processing process:

4.1. Image Grey Scale Processing

The primary purpose of image grayscale processing is to gather colour images of apples. The weighted average method is used in this procedure to convert 24-bit images to 8-bit images. The formula is:

\[\label{e1}\tag{1} {{C = xR + yG + zB, x + y + z = 1}},\] where \(C\) is the colour after conversion, \(x\), \(y\), and \(z\) are the three most fundamental colour coefficients, and \(R\), \(G\), and \(B\) are the three primary colours.

4.2. Normalisation

The apple image that needs to be detected can be normalised during image processing to give it a consistent size. Assuming that the scaling ratios in the X- and Y-axes directions are \({f_x}\) and \({f_y}\), respectively, normalisation takes the form of uniform image scaling. The expression for the point (\({x_1},{y_1}\)) in the scaled image \(g(x,y)\) corresponding to a point (\({x_0},{y_0}\)) in the original apple image to be recognised is \(f(x,y)\).

\[\label{e2}\tag{2} {x_1} = {f_x} \times {x_0},{y_1} = {f_y} \times {y_0}.\]

As feature points are rearranged within the range of [0, 255], the normalisation procedure can increase the accuracy with which the apple image is detected. The normalisation formula for the grey scale is:

\[\label{e21}\tag{3} p(i,j) = {{p(i,j) – {G_1}} \over {{G_2} – {G_1}}} \times 255.\]

4.3. Image Enhancement

Additional image processing can be used to increase the apple photos’ identification efficacy. This may involve filtering operations to eliminate interference from the image, as well as operations such as dilation and erosion to enhance the image’s resolution, thereby improving the accuracy of apple grading [18].

4.4. Image Binarization

For convenient image processing and enhanced recognition efficiency, binarization of the image is performed by setting the grayscale values to 0 or 255. Niblack algorithm can be used to achieve binarization [19]. This algorithm determines the central coordinates \((x,y)\) on the apple image to be identified and calculates pixel variance and mean within the surrounding neighborhood of size \((r*r)\) facilitating the binarization process.

\[\label{e3}\tag{4} T(x,y) = m(x,y) + k \times s(x,y),\]

\[\label{e4}\tag{5} m(x,y) = {1 \over {{r^2}}}\sum\limits_{i = x – {r \over 2}}^{x + {r \over 2}} {\sum\limits_{j = y – {r \over 2}}^{y + {r \over 2}} f } (i,j),\]

\[\label{e5}\tag{6} {\rm{s}}(x,y) = \sqrt {{1 \over {{r^2}}}\sum\limits_{i = x – {r \over 2}}^{x + {r \over 2}} {\sum\limits_{j = y – {r \over 2}}^{y + {r \over 2}} {[f} } (i,j) – m(x,y){]^2}} ,\] where \({\rm{T}}(x,y)\) is the threshold value of pixel \((x,y)\); \(m(x,y)\) is the mean value of pixel in \((r*r)\) neighbourhood; \(s(x,y)\) is the standard deviation of pixel in \((r*r)\) neighbourhood; \(k\) is the correction coefficient, which can be binarised to set pixel grey scale to 0 or 255.

4.5. Feature Dissection

Intersection points and endpoints are crucial to the matching process in the recovered texture skeleton picture of the apple image [21]. The specific algorithm for extracting feature points is assuming that the \(3*3\) template region is:

\[\begin{gathered} \begin{matrix} {{P_1}} & {{P_2}} & {{P_3}} \\ {{P_g}} & {{P_0}} & {{P_4}} \\ {{P_7}} & {{P_6}} & {{P_5}} \\ \end{matrix} \quad \end{gathered}\tag{7}\] where \({P_1}\) represents the centroid of the region and its formula can be expressed as follows:

\[\label{e7}\tag{8} N_{\text {trans }}=\sum_{i=1}^g\left|p_{i+1}-p_i\right|,\] where \(P_1=P_9\) in this case. Once the image’s features have been extracted, the apples can be ranked according to the diameter, area, and edge features of the image. After classifying each grade into pertinent standards, the apples are sorted based on the outcomes of the image recognition process.

5. Identification System Testing

The most important part of the apple sorter is the picture recognition system, which is based on the PLC control system’s recognition results for automatic apple sorting [14]. This time, as seen in Figure 5, we primarily test the sorter’s image recognition system using apples as the test object [23].

Figure 5. Selected Apples Schematic

The feasibility of the recognition system was assessed by selecting apples as the identification objects, focusing on four apples with clearly distinct features for sorting. Initially, one of the apples underwent image processing, and the result after gray scale processing is depicted in Figure 6.

Figure 6. Image After Grey Scale Processing

Apple diameter and circumference can be extracted based on contour data. This time, as illustrated in Figure 7, the primary method of classification for apples is the circumference of their outlines. Four apples need to be categorised.

Figure 7. Image of Apple to be Graded

The perimeter of the contours created after the apple photographs were processed was the main grading factor for the four apples that were graded. The grading outcomes, which are displayed in Table 1, were produced by processing the four apple photos and counting the perimeters.

Table 1: Apple Grading Level Table
Apple signal to be tested Measured circumference/cm Rating
1 225 3
2 315 3
3 350 2
4 420 1

Table 1 clearly shows that the perimeter data of apples can be obtained by effectively using the specified image processing approach. Through empirical grade assessment, the quality grade of apples can ultimately be determined. In the actual operation of the apple sorting machine, after obtaining the assessed grades, To categorise various apples and transport them to separate sites for packaging and processing, the control system must be used.

Figure 8. Pipeline Orange Colour Grading Detectiond

Figure 8 shows the detection effect and classification detection effect of oranges. To achieve a better recognition effect, a binarized image is used as an auxiliary. Binarized images are frequently used in image processing and computer vision due to their simplicity and utility. They help to simplify data, extract features, reduce noise, segment, and support thresholding. Binarization may assist in data compression and preprocessing for complex computer vision tasks. However, it may result in some information loss, making the selection of the binarization method critical, and its application is dependent on task-specific requirements. After using the background subtraction method to remove the background, according to the RBG and HIS (Hue, Saturation, Intensity) conditions, determine the area where the orange appears, and select it immediately. Using the background subtraction method, a computer vision algorithm detects objects in video content [8].

The algorithm compares objects to the image’s background and foreground. During the no-object-frame comparison, the objective is to isolate foreground objects. The background subtraction method is an effective technique for detecting moving objects in a series of frames from static cameras. The method detects foreground objects by comparing incoming frames to a model of the scene’s background. The background subtraction method is generally fast and requires less power for computation. Figure 7 shows the original greyed-out image on the top left, the RPN output effect on the top right, the RGB mapping map on the bottom left, and the output map of the entire model—that is, the discrimination of the fruit levels—on the bottom right. Different colours correspond to different levels.As can be seen from Figure 7, compared with the traditional L/C (brightness/contrast) method, the orange detection effect of the algorithm in this paper is better.

Therefore, after comprehensive consideration, the average hue is used to classify the red-orange colour, in which the average hue of the superior grade is less than or equal to 26, and the average hue of the first- and second-class grades is 18 and 13, respectively. Therefore, the specific grading parameters of the tone mean value among each grade are shown in Table 2

Table 2: Grading Parameters of Hue Mean Value Between Grades
Superior products First class Second class Substandard
Mean value \(\le\) 26 13\(<\)mean value \(\le\) 18 8\(<\)mean value \(\le\) 13 mean value<8

6. Conclusions

Machine vision technology has been included into the apple sorting system’s design to increase apple sorting efficiency and enable non-destructive testing. Through processes such as grayscale treatment, binarization, enhancement, and feature extraction on apple images, automated grading of apples has been achieved based on established grading standards. The apple sorting machine’s image recognition technology was tested in order to confirm the solution’s viability. The findings show that the sorting image recognition system can effectively classify grades and conduct non-destructive testing by sorting apples according to their perimeter features. This holds great significance as a point of reference for the advancement of automated fruit sorting mechanisms.

Funding

This work no funding supported.

Conflict of interest

The author declares no conflict of interests.

References:

  1. Patel, K.K., Kar, A., Jha, S.N. and Khan, M.A., 2012. Machine vision system: a tool for quality inspection of food and agricultural products. Journal of Food Science and Technology, 49, pp.123-141.
  2. Cubero, S., Aleixos, N., Moltó, E., Gómez-Sanchis, J. and Blasco, J., 2011. Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables. Food and Bioprocess Technology, 4, pp.487-504.
  3. Chaudhari, D. and Waghmare, S., 2022, May. Machine vision based fruit classification and grading—a review. In ICCCE 2021: Proceedings of the 4th International Conference on Communications and Cyber Physical Engineering (pp. 775-781). Singapore: Springer Nature Singapore.
  4. Narendra, V.G. and Hareesha, K.S., 2010. Quality inspection and grading of agricultural and food products by computer vision-a review. International Journal of Computer Applications, 2(1), pp.43-65.
  5. Zhang, W., Zhao, B., Gao, S., Ji, Y., Zhou, L., Niu, K., Qiu, Z. and Jin, X., 2023. Online Recognition of Small Vegetable Seed Sowing Based on Machine Vision. IEEE Access, 11, pp.134331-134339.
  6. Yang, K., Liu, H., Wang, P., Meng, Z. and Chen, J., 2018. Convolutional neural network-based automatic image recognition for agricultural machinery. International Journal of Agricultural and Biological Engineering, 11(4), pp.200-206.
  7. Duro, D.C., Franklin, S.E. and Dubé, M.G., 2012. A comparison of pixel-based and object-based image analysis with selected machine learning algorithms for the classification of agricultural landscapes using SPOT-5 HRG imagery. Remote Sensing of Environment, 118, pp.259-272.
  8. McCarthy, C., Hancock, N. and Raine, S.R., 2010. Apparatus and infield evaluations of a prototype machine vision system for cotton plant internode length measurement. Journal of Cotton Science, 14(4), pp.221-232.
  9. Mahendran, R., Jayashree, G.C. and Alagusundaram, K., 2012. Application of computer vision technique on sorting and grading of fruits and vegetables. J. Food Process. Technol, 10, pp.2157-7110.
  10. Mahirah, J., Yamamoto, K., Miyamoto, M., Kondo, N., Ogawa, Y., Suzuki, T., Habaragamuwa, H. and Ahmad, U., 2017. Monitoring harvested paddy during combine harvesting using a machine vision-Double lighting system. Engineering in Agriculture, Environment and Food, 10(2), pp.140-149.
  11. Patrício, D.I. and Rieder, R., 2018. Computer vision and artificial intelligence in precision agriculture for grain crops: A systematic review. Computers and Electronics in Agriculture, 153, pp.69-81.
  12. Ebrahimi, E., Heydari, M., Bavandpour, M. and Heydari, H., 2014. Presenting a Machine Vision System in Application of Wheat Sorting. Asian Journal of Research in Business Economics and Management, 4(3), pp.383-390.
  13. Ramezani, H., Khoshtaghaza, M.H., Minaei, S. and Akbarizadeh, G., 2023. Machine Vision System for Autonomous Guidance of Raisin-Collection Rover. AUT Journal of Mechanical Engineering, 7(1), pp.6-6.
  14. Chung, C.L., Huang, K.J., Chen, S.Y., Lai, M.H., Chen, Y.C. and Kuo, Y.F., 2016. Detecting Bakanae disease in rice seedlings by machine vision. Computers and Electronics in Agriculture, 121, pp.404-411.
  15. Hassankhani, R. and Navid, H. 2012. Quantitative sorting of potatoes by means of machine vision system. International Journal of Agriculture: Research and Review, 2(4), pp.443-447.
  16. Brosnan, T. and Sun, D.W., 2004. Improving quality inspection of food products by computer vision––a review. Journal of Food Engineering, 61(1), pp.3-16.
  17. Irumva, T., Mwunguzi, H., Pitla, S.K., Lowndes, B., Yoder, A.M. and Siu, K.C., 2023. Agricultural Machinery Operator Monitoring System (Ag-OMS): A Machine Learning Approach for Real-Time Operator Safety Assessment. Journal of Agricultural Safety and Health, 29(2), pp.85-97.
  18. Ali, J., Shan, G., Gul, N. and Roh, B.H., 2023. An Intelligent Blockchain-based Secure Link Failure Recovery Framework for Software-defined Internet-of-Things. Journal of Grid Computing, 21(4), p.57.
  19. Ren, X., Ahmed, I. and Liu, R., 2023. Study of Topological Behavior of Some Computer Related Graphs. Journal of Combinatorial Mathematics and Combinatorial Computing, 117, pp.3-14.
  20. Nazeer, S., Sultana, N. and Bonyah, E., Cycles and Paths Related Vertex-Equitable Graphs. Journal of Combinatorial Mathematics and Combinatorial Computing, 117, pp.15-24.
  21. Kalantari, D., Jafari, H., Kaveh, M., Szymanek, M., Asghari, A., Marczuk, A. and Khalife, E., 2022. Development of a machine vision system for the determination of some of the physical properties of very irregular small biomaterials. Int. Agrophys, 36, pp.27-35.
  22. Khan, R.U., Khan, K., Albattah, W. and Qamar, A.M., 2021. Image-based detection of plant diseases: from classical machine learning to deep learning journey. Wireless Communications and Mobile Computing, 2021, pp.1-13.
  23. Chen, Y.R., Chao, K. and Kim, M.S., 2002. Machine vision technology for agricultural applications. Computers and Electronics in Agriculture, 36(2-3), pp.173-191.