I'm not sure if it wasn't actually asleep. I just click the Apple menu > Sleep. Leave it. Go outside. Trigger my wake macro remotely, and it wakes up so I can access it via AnyDesk. Like @peternlewis mentioned above, there might be different levels of sleep.
I'll check on "Wake for network access". I'm not good at network though and I thought its only for LAN.
Didn't thought of putting a SwitchBot on the Keyboard That's smart.
I can leave the mac mini running without making it sleep. Stupid question, does the lock screen due to inactivity make my device sleep?