Recently, there has been a growing interest in polymer insulating materials that incorporate nanoscale inorganic additives, as they have shown significantly improved dielectric, thermal, and mechanical properties, making them highly suitable for application in high-voltage insulating materials for electrical machines. This study aims to improve the dielectric and thermal properties of a commercial polyester varnish by incorporating different concentrations of titanium dioxide nanoparticles (TiO2) with proper surface functionalization. Permafil 9637 dipping varnish is the varnish used for this investigation, and vinyl silane is the coupling agent used in the surface functionalization of TiO2 nanoparticles. First, nanoparticles are characterized through Fourier transform infrared spectroscopy to validate the success of their surface functionalization. Then, varnish nanocomposites are characterized through field emission scanning electron microscopy to validate the dispersion and morphology of nanoparticles within the varnish matrix. Following characterization, varnish nanocomposites are evaluated for thermal and dielectric properties. Regarding thermal properties, the thermal conductivity of the prepared nanocomposites is assessed. Regarding dielectric properties, both permittivity and dielectric losses are evaluated over a wide frequency range, starting from 20 Hz up to 2 MHz. Moreover, the AC breakdown voltage is measured for varnish nanocomposites, and the obtained data are incorporated into a finite element method to obtain the dielectric breakdown strength. Finally, the physical mechanisms behind the obtained results are discussed, considering the role of nanoparticle loading and surface functionalization.