That depends a great deal on how you get your aircraft. If you build a new aircraft from plans, careful shopping can keep the cost at or a bit below $5000. You will take some time getting it built, but it is a tremendous satisfaction to fly an aircraft you have built from scratch!
Most ultralight gyroplane kits will come in between $6000 and $10,000. Building time will vary from model to model, but it is typically very short compared to a plans-built machine since most (or all) of the critical drilling and cutting has been done for you
Buying a used machine that is already ultralight-legal or which can be modified to be may be the most economical approach, but you do need to be careful about what you are getting - especially if the price seems too good to be true! Your budget should be flexible enough to handle the costs of repairing, upgrading, or otherwise modifying the machine. Assuming you are new to the sport, it would be a good idea to have a used machine thoroughly inspected by a qualified gyro pilot prior to purchase. This way you can avoid unpleasant surprises.