As I framed this scene, the woman in blue scrunched up her face and tried to turn her body. As she passed, she said angrily that she didn’t appreciate having her photograph taken.

In the United States, we have the right to photograph anything we can see when we stand in public spaces (like this sidewalk). The American Civil Liberties Union elaborates this and other rights photographers have here.
I had the right to photograph this woman, whether she liked it or not. But if I had any idea of her feelings I would have waited until she had passed me by. I wanted to capture this sidewalk with pedestrians walking along it, but this particular woman did not have to be included. Other people would have come along who would not have objected, possibly would not have even noticed me. Why needlessly upset people?
I also have the right to photograph children I see in public spaces, but I seldom do. When my children were young I would have objected if some strange man photographed them at the playground. One possible reason a man might do that is unsavory enough that, even though honorable reasons are far more likely, I would have gotten in his face and insisted he leave immediately.

However, as you can see above, children do sometimes appear in my photographs. In this case the scene was the busy National Mall in Washington, DC. As a popular tourist destination, many people in the crowd were making photographs. It allowed me to be inconspicuous. Still, I didn’t linger. I wanted to photograph a rider on the dragon, the most interesting “horse” on the Mall’s carousel. I also wanted to practice panning to stop in-motion subjects. I made three quick photographs, two on one pass and one on the next, and then turned my attention to other subjects.