During breast cancer progression, there is typically increased collagen deposition resulting in elevated extracellular matrix rigidity. This results in changes to cell-matrix adhesion and cell migration, impacting processes such as the epithelial-mesenchymal transition (EMT) and metastasis. We aim to investigate the roles of cell-matrix adhesion and cell migration on breast tumor growth and progression by studying the impacts of different types of extracellular matrices and their rigidities. We embedded MCF7 spheroids within three-dimensional (3D) collagen matrices and agarose matrices. MCF7 cells adhere to collagen but not agarose. Contrasting the results between these two matrices allows us to infer the role of cell-matrix adhesion. We found that MCF7 spheroids exhibited the fastest growth rate when embedded in a collagen matrix with a rigidity of 5.1 kPa (0.5 mg/mL collagen), whereas, for the agarose matrix, the rigidity for the fastest growth rate is 15 kPa (1.0% agarose) instead. This discrepancy is attributable to the presence of cell adhesion molecules in the collagen matrix, which initiates collagen matrix remodeling and facilitates cell migration from the tumor through the EMT. As breast tumors do not adhere to agarose matrices, it is suitable to simulate the cell-cell interactions during the early stage of breast tumor growth. We conducted further analysis to characterize the stresses exerted by the expanding spheroid on the agarose matrix. We identified two distinct MCF7 cell populations, namely, those that are non-dividing and those that are dividing, which exerted low and high expansion stresses on the agarose matrix, respectively. We confirmed this using Western blot which showed the upregulation of proliferating cell nuclear antigen, a proliferation marker, in spheroids grown in the 1.0% agarose (≈13 kPa). By treating the embedded MCF7 spheroids with an inhibitor or activator of myosin contractility, we showed that the optimum spheroids' growth can be increased or decreased, respectively. This finding suggests that tumor growth in the early stage, where cell-cell interaction is more prominent, is determined by actomyosin tension, which alters cell rounding pressure during cell division. However, when breast tumors begin generating collagen into the surrounding matrix, collagen remodeling triggers EMT to promote cell migration and invasion, ultimately leading to metastasis.