This study investigates the effect of using a hybrid nanofluid filter on the performance of a PV system. The ZnO-MgO/Water-Ethylene Glycol (W-EG)-based filter presents an economic and green optical filter. This study compares the effects of using a hybrid filter against other filters under a variable optical fluid mass flux (0-10 kg/s.m^2), a maximum radiation intensity (1000 W/m^2), and a fixed filter thickness (10 mm). The results show that a dynamic W-EG (60:40) filter with mass flux (1-6 kg/s.m^2), can reduce the temperature of the PV cells by 45% of that for a stationary filter. In addition, the performance of the spectral splitting filter-PV (SSF-PV) system shows an average enhancement ratio in the electrical efficiency by 28% more than that of the reference PV panel, and an average improvement in the cooling effect by 26%, relative to the fixed inlet temperature (20℃) of optical fluid. The proposed nanofluid-based SSF-PV system achieves an optimum electrical efficiency and enhanced cooling effect compared with investigated nanofluids.