I bought a beautiful pinkish russet old western dress hat made in Wichita Falls, Texas, for only a dollar at a used store in Mason, Texas, because it had several moth holes in it about the diameter of a toothpick or match stick. I called the company and they said worm holes couldn't be successfully repaired by the glue and felt-scrapings method because the patch color wouldn't match the sun faded color of the rest of the hat surface.
Not to be discouraged, I successfully filled the holes with "plugs" made by trimming a matchstick-
wide strip from the brim so I could cut plugs from it. I used sharp titanium scissors, jewelers magnifying glasses, some slow-drying Elmer's glue, tweezers, a tiny jeweler's screwdriver or something similar as a plug punch, and a steady hand.
With the scissors I cut a slightly oversized plug (the felt will compress) from the removed brim strip, the weathered area to face outward, matching the weathered hue of the hat. Matching the exposed surface of the plug to the outer color of the hat is most important if you want the repair to be invisible. Some slow drying glue is then smeared in the bottom areas of the hole with the tiny screwdriver or a toothpick and the plug inserted with tweezers. Mash the plug in with the flat of the screwdriver until the plug surface aligns with the surface of the hat. The glue at the bottom will hold the plug in place. Stay away from fast-drying glue so you can manipulate and adjust.
After all holes are filled rough up and round off the scissored surface of the brim where the strip was removed. I used a diamond file but sandpaper should work just fine.
I now have a new looking hat costing a buck and about 45 minutes worth of effort. You will get better at it with practice. Felt scrapings into glue will always show darker than the rest of the hat. a properly inserted and color matched plug won't.
Good luck.
Texan Forever
August 2014