How much RAM do you have and how much did you allow QuPath to use when you started it?
Some projects will likely be too big for certain computers if you want to analyze millions of cells. In some cases, if you are capping out on RAM, you might be able to reduce the number of processors used in the Preferences, or increase the amount of RAM available in the Help menu. I would start there, with the most likely to work option being turning the processors down to 1. It might take a very long time, though. That is the tradeoff.
Hard to help too much without more information, like the exact error, what your computer specs are, how many cells/what area is “big.”
If your computer is not strong enough and your cannot turn down the settings any further, you may want to look into subdividing your analysis. You could tile your tumor area, and then run the cell detection one smaller tile at a time. The smaller your tiles, though, the more cells you will split in half by having an annotation border run through them.