We had both CX-3 and CX-5 in my garage at the same time. CX-3 was mine, CX-5 is my wife's. Both 2016 models. For my needs (family of 4, two kids). CX-3 was okay most of the time. 90% of the time it was just me in the car commuting to/from work. I didn't require the extra space most times. I could still get all four of us in the car without too much trouble because my kids are still small enough, they can get by with less leg room. We had the CX-5 for times were we needed more cargo room.
Here's some cases where I would NOT get the CX-3.
1. If there's any chance you might have a baby in the time you'll own the car. Getting a rear-facing baby seat in the back will not be easy without putting at least one of the front seats quite far forward.
2. I used to take guitar lessons and my Les Paul guitar would not fit in the cargo area without folding at least one of the back seats down. I instead usually just set it in the front seat foot area with the neck resting against the seat base (I have it in a hard case). I got by this way most of the time, but it was a bit annoying. This was just a one-day per week situation. Not something I wanted to get rid of the car over, but some people may find this a bigger problem than I did. If you anticipate a same or similar situation, go for the CX-5.
3. I'll second Anchorman's comments on build quality and cabin noise. The CX-3 developed a bunch of minor, but annoying, rattles that I never took the time to sort out. It didn't give some of my co-workers a great impression when a new car was making so many noises when we'd go to lunch. Our CX-5 has developed a few minor noises as well, but it took more miles for it to happen, and I think at least some of that is from all the assorted random junk my wife and kids keep in her car at all times.
4. I was surprised to find that my wife gets nearly the same fuel economy in her CX-5 (FWD) as I got in my CX-3 (AWD). I was only getting maybe 1-2 MPG better fuel econ, though I was also doing so with AWD vs. her FWD. I was anticipating a little better fuel econ than I got. If I'd known when I bought it that the fuel econ was going to be basically the same, this might have weighed on my purchasing decision a little heavier. I found that I was often pushing the gas pedal harder on the CX-3 to get it to move. Our CX-5 requires less of the right foot to get it moving.
5. Consider the 2019 Mazda 3. Will be out early next year if you can wait a few months. They're allegedly offering an AWD version, not sure if it will be available at launch or if they will delay it to coincide with the release of the SkyActive-X engine which might not be out until a mid-year update. Having AWD was, to me, the only advantage the CX-3 had over the Mazda 3. The 3 is otherwise better in almost every way, at least on paper. About same price. 3 is a bit bigger inside and out. Almost the same ground clearance. More HP and somewhat better real world fuel econ despite the bigger engine. If an AWD 3 was available in 2016, it would have been a no-brainer for me to buy the 3 instead of the CX-3.
6. BOTH our CX-3 and CX-5 had failures of the entertainment system computers. Same failure mode where the system gets stuck in a boot loop and eventually fails completely. Mazda replaced the CMU's on both cars under warranty. We got lucky with the CX-5 because it failed just a month before extended warranty we bought expired due to mileage. That would have been over a $1000 repair, so the warranty was worth the cost in our case. This was enough to have me second-guessing about buying Mazda again when I had to get another car earlier this year after my CX-3 got totaled. Do I want to take another chance on the Mazda infotainment system? We're 2-for-2 now on this MazdaConnect system in having total failures. It should be noted that the CX-3 and CX-5 we had were built within about a month of each other, so it's possible they both had CMU's from the same batch of bad parts from the manufacturer. Ultimately, I ended up picking up a 2018 Mazda 6 GT on a lease.
Overall, we've been happy with both the CX-3 and CX-5. I wasn't happy enough with the CX-3 to get another one when it came time to replace mine. I went with the 6 instead. In retrospect, it probably would have been better to get the 6 back in 2015. With all the aggressive incentives they were offering at the time, the 6 would have been a couple thousand dollars cheaper than the shiny new CX-3 at the time and I would have saved some money on gas along the way too. AWD was only ever a significant benefit on a handful of days during my ownership. If I'm still enjoying my 6 at the end of my lease, I'll probably buy it out. I leased it in April and so far, I'm very happy with it. The only thing I don't like is the mileage restrictions of my lease, or I'd be out enjoying it even more! I should have negotiated more miles/year on the lease terms. I was planning to use my wife's CX-5 on more of our driving than I have been. At the rate I'm going, I'll be over my limit by around 1-2K miles per year.