Router insert plate?

On my home made router table the home made insert plate that holds the router the centre cutter hole is inch and half across. I've been told it is considered dangerous to use smaller cutters [smallest used 6mm straight cutter] without inserting a hole insert plate to reduce the size of the hole,...true or false?.

Any router experts out there who can advise. I have been using it for some time with no effect on safety. I can't see why using the manufactured insert hole reducers makes any difference plus I can't rebate the hole unless I stick a disc on the underside to form a rebate.