I may be feeding a troll but here goes anyway.
So long as I'm not hurting anyone else, what right of it is yours to tell me that I may not do something?
Because you are part of a community you selfish prick.
In a community you do stuff that costs you in order to make a stronger community. You pay part of your income to the city/state/country/whatever. In return, fire engines come to your house, and to the houses of other members of the community. The roads are paved and level. The water and electricity get to your house. The police arrest criminals before they break into your house and steal your stuff. The country's military and intelligence agencies stop many threats to the country before those threats are realised. There is a pretty good chance that when you wake up tomorrow morning the government will be the same as it was when you went to bed.
If you have more money than your neighbours then yes, you help them! You provide for their housing and feeding (either directly through charity or indirectly through taxes and/or tithes). If you own a business then yes, you pay the money to install a wheelchair ramp. Why? First, it means that when life shits on you then someone else provides for you. Second, when those people get out of the bad situation they contribute to the society themselves, making the whole society stronger. Third, because it is the decent human thing to do!
EvilLegion, you sound like one of those rich, healthy, materialistic "libertarians" for whom libertarianism simply means "I get to do what I want and the poor can go fuck themselves". I call you on the strength of your convictions and challenge you to go an live in a place where that attitude really applies. I suggest northern Afghanistan or Somalia or many other places in the world where law and order has broken down and the social contract no longer applies.
Short answer: Can I provide one shred of reason why it is your responsibility? Yes. It is because acting that way is the sign of a responsible member of a community. If you are not willing to act that way and contribute to the community then you shouldn't be accepting the benefits of being part of the community.
I'm not normally a fan of pithy sayings but sometimes they are appropriate. Freedom of speech does not give you the right to yell 'fire" in a crowded theatre.