In 1919, the Girl Scout Association of the District of Columbia began serving tea and refreshments to people visiting the reclaimed mud flats at the convergence of the Potomac and Anacostia Rivers known as Hains Point. Visitors were drawn to the triangular island by the calm of the riverbanks, the views of Virginia across the water, and, in the spring, the flowering cherry trees planted around the perimeter. They could watch "spectacular aerial stunts" by pilots from the naval station across the Washington Channel. People often arrived via the new road around the island known as the "Speedway." The automobile was an innovation that was just becoming more accessible to the public. People out for a drive around the nation's capital found themselves in need of a comfort station and sustenance out at Hains Point, but the nearest one was three miles away.

At first, the Girl Scouts staffed a temporary shelter with tables where they served tea, cold drinks, ice cream, and snacks to travelers. The stand became so popular that even President and Mrs. Harding regularly stopped by. By 1922, the Office of Public Buildings and Grounds asked Congress to provide funding for a permanent structure.

The Girl Scouts moved into the new building in 1924. The portico of the pavilion with its five double-french doors topped with fanlights were reminiscent of the porch at Mount Vernon; opening the doors allowed the breeze blowing across the river to cool the interior in the time before air conditioning. Visitors could relax on the front porch or on a patio out back, far from the noise of the city. President Calvin Coolidge and his wife Grace were among the customers who came to enjoy the river views and a cool beverage.

But the next year, a new Office of Public Buildings and Public Parks of the Nation's Capital replaced the Office of Public Buildings and Grounds. The new director, Lieutenant Colonel U.S. Grant III refused to renew the Girl Scouts' permit to operate the tea house which expired at the end of 1925. A new concessionaire was appointed to operate the facility. That company, which became the present-day Guest Services Incorporated (GSI), continued to operate the teahouse and offer food and beverage service, including wine and beer beginning in the 1930s. The facility operated during three seasons and closed during the winter. It was especially popular when the cherry blossoms appeared during the early spring.

In 1933, Washington DC's parks—including Hains Point—became part of the National Park Service. The Tea House continued to operate under the same concessionaire. It closed during World War II because gasoline and rubber for tires were in short supply, so driving the Speedway was discouraged. A fire damaged the snack bar in 1949; with the insurance funds, the building was renovated to include an expanded kitchen. GSI changed the name of the location to the Hains Point Inn in 1950. Several different subconcessioners ran the Hains Point Inn from 1952-1957, when the name was changed again to the Cherry Blossom Inn.

The Inn closed by 1962 and reopened as a National Park Service visitors' center until 1967. For nearly two decades, the old Teahouse was the Ecological Services Lab of the National Capital Parks. It fell into disrepair after a large flood of Hains Point in 1985. Although a Historic American Buildings Survey prepared in 1987 noted its historic significance, the Teahouse was demolished soon after.