Microgels are colloidal-scale particles individually made of cross-linked polymer networks that can swell and deswell in response to external stimuli, such as changes in temperature or pH. Despite many experimental studies on microgels, an accurate theoretical description based on individual particle properties is still missing due to the complexity of the particles. To go one step further, here we propose a novel methodology to assemble realistic microgel particles in silico. We exploit the self-assembly of a binary mixture composed of tetravalent (cross-linkers) and bivalent (monomer beads) patchy particles under spherical confinement to produce fully bonded networks. We then use the resulting structure to generate the initial microgel configuration, which we subsequently simulate with a bead–spring model complemented by a temperature-induced hydrophobic attraction. We show that we can reproduce the experimental swelling curve by appropriately tuning the confining sphere radius, which would not be possible with less sophisticated assembly methodologies, e.g., in the case of networks generated from an underlying crystal structure. We further investigate the structure, in reciprocal and real space and the swelling curves of microgels as a function of temperature, finding that the widely used fuzzy sphere model well describes our results.