The article firstly establishes a mathematical model of the FMS shop floor planning process problem, and combines the rescheduling strategy and rolling scheduling strategy for solving the FJSP problem. Subsequently, the simulated annealing genetic algorithm is improved by relying on genetic algorithm, simulated annealing algorithm and particle swarm optimization algorithm, and the application of hybrid optimization algorithm in problem solving is proposed. The simulated annealing algorithm is incorporated into the crossover and mutation operations of the genetic algorithm to strengthen the local search capability, and then the global annealing operation is incorporated into the new individuals obtained. The overall design of the mixed reality-based FMS virtual simulation system was tested with a view to optimizing the external tool library tool limitation problem in the FMS shop floor planning process. The results of the simulation experiments show that although the algorithm of this paper, SaDE and CoDE algorithms can reach the optimal solution, the convergence speed of the algorithm proposed in this paper is significantly better than the other two algorithms. Based on the experimental results, the article finally constructs a mixed reality-based FMS virtual simulation system to solve the external tool library tool limitation problem in the FMS shop floor planning process.