Most museums contain a library available to the public. It may be one room in a small local museum or a large collection of books and manuscripts. Generally anyone may research their holdings, although a couple, the Folger (DC) and Huntington (CA) have restrictions.
Colonial Williamsburg has a two story library with a separate reading room for manuscripts, ledgers, rare books, etc. The Peabody Essex Museum library in Salem also contains manuscripts, ships journals and more in a centuries old building.
Occasionally, the library may not be in the facility. The Mariners’ Museum in Newport News, VA no longer maintains their library in the one end of their building, but has moved it to the Christopher Newport University. The Smithsonian has numerous libraries in its various museums, and even separate buildings.
If you are interested in visiting a museum library and its museum check their website or call for hours and any restrictions. Most have limited hours and are closed on weekends.