"How do I stop this? I know how to stop my primary thoughts, but not this background chatter, the chatter that is merely a nuisance to my meditation and doesn't distract me away from my object. "
The very idea that thoughts are an obstacle to your meditation in itself is what is creating so much unecessary struggle. Thought is not a hindrance to one's meditation, neither are emotions, perceptions, or any number of things which may be happening in one's experience. The problem is that you are unable to remain a witness as to whatever arises in your experience without becoming identified. The moment one has become identified, one loses all clarity. What is needed is an eye which is capable of seeing things with neither attraction or aversion, without judgement, without liking or disliking, raising a finger for or against. Once you can remain a witnessing awareness without clinging to what arises in your experience, then the mind naturally starts to settle itself - it comes closer and closer to silence. But if you are struggling with your thoughts, then it is just like an ox chasing one's own tail - or trying to strike the lake in order to bring it to stillness.
The second problem is that you are assuming that what is happening is meditation. Meditation is not focusing the mind on a particular object, nor is it something that can be done as an effort of the mind. Concentration can be done as an effort of the mind. And you can use concentration as a means to enter into meditation, but focusing the mind on an object is not meditation.
"At first it was very easy for me to keep my mind completely silent, while holding my breath."
Then you have not been watching your experience closely. Otherwise, you would know very well that this is not silence. If your mind was silent - even the very intention and effort to do something like hold your breath would not exist.
"I do remember reading that advanced yogis, such as Swami Rama can dive into the depths of their subconscious with ease to retrieve thoughts, so could it be possible that this is just the start of me entering that terrain?"
As one's awareness turns inwards, various things which lie beneath the surface of your conscious mind will start surfacing. This will take the form of various dream-like hallucinations. Do not be alarmed if such things happen - just continue remaining a witness as to whatever is arising. Eventually, even these experiences will subside until the mind becomes absolutely silent.