Yes, I believe a similar sort of afflication affects me, for I too do have some difficulty understanding some mathematical concepts, for I cannot apply them while contained in their abstract state to "real world" situations, unless they are visualised somewhat by me. That is why I could only really gain a full understanding of Vector Analysis when I applied it to Electrodynamics (well, classical in that case), and the abstract notation is made rational, as are continuous vector field concepts, such as of course the Electric or Magnetic fields, and interactions thereof. Though that's just vectors, I hope this example illustrate what I mean, the similar case for calculus when I was learning it, though this time Newtonian dynamics. Why, when I was little (about 10), I had limited mathematical understanding and tried to "make sense" of things such as electrodynamics (otherwise I would generally not accept anything to do with it). I ended up thinking of charges being tempestuous defects on the matter of the objects themselves (then of course having no knowledge of electrons, protons and the like), which sufficiently neutralised once a corresponding system of defects on the matter of the other charge was brought near. However, I was then stuck with another problem, which was how the damn things interacted in the first place, which was more difficult for me to do. Later I attributed this to great arcs and lances projecting from the two charged matters towards each other, rather like images of the Sun I saw when I was a child, permeating space, and so the projections would act as "conduits" for the "charge" and would interact as such with a secondary matter object. But then, as I learned of electron theory and the like, I used my visualisation view of electrodynamics less, but still, I could not find any satisfying explanation for how charge exists (let alone comes about) in the first place, or even matter itself, and that, among many other things, still keeps me in wonderment to this day.
I myself am not an avid viewer of science fiction, for there are still sources of amazement within our own Universe, still many unsolved mysteries and things to be explored. Why should one dwell in an imaginary world, when the real world is more spectacular still? However, one must still dwell in an imaginary world even if we wish to dwell in this "real" universe, for all knowlege is simply created by people agreeing on their views of how the Universe fundametally operates, of course localised within some sort of a mathematical framework, as a "theory". And thus even physics is trapped in this imaginary world, where only the human mind itself impedes knowlege and understanding, that itself a realm contained within the human consciousness. Although, in my case, I just choose to designate one view as "real" the other "imaginary", but in reality both are to some extent "imaginary". Though since they are affected by real mechanisms operating around us manipulating matters within us intrinsic to our reality, one may say imagination is incredibly petty, for it is merely a manifestation of "real" processes, severely limiting the human desire for a true "imaginary" picture to be painted in one's mind, for it is all real to begin with, and shall be for all eternity, in space, time, and matter incarnate, as far as we know in our picture anyway.