OK, I am not sure why but I was able to fix it by switching the order of my divide_by in the calculation.
Example
{{ settings.font_number_size | times: settings.font_default_scale | divided_by: settings.font_body_size | divided_by: 100.000 | round: 3 }}
To
{{ settings.font_number_size | times: settings.font_default_scale | divided_by: 100.000 | divided_by: settings.font_body_size | round: 3 }}
Well, it resolves my specific issue
I looked at this again and to be easier just adding a times or divide_by as 1.000 fixes it too.
For example
{{ settings.font_number_size | times: settings.font_default_scale| times: 1.000 | divided_by: settings.font_body_size | divided_by: 100 | round: 3 }}