So, the rubber adapter is smaller than the secondary manifold?
If so, please disregard my advice, I thought it was the other way. Not 100% ideal but much better than the other way.
In this case, simply build up the ports with 2-part epoxy (JB Weld) and use sandpaper/dremel to make a smooth transition from the adapter to about half-way along the secondary manifold.