In reply to Patrick Surguy:
Def not needed for Hoy and you can save quite a bit of money by going on foot/bike. As others have said, there is a taxi on Hoy that will pick you up from the Stromness - Moaness ferry and take you to Rackwick (and back).
There is a lovely walk round the coast from Stromness to Yesnaby (taking in N Gaulton castle on the way), it's maybe 8 miles or so total but if you are stopping at N Gaulton then slightly shorter the first day and then a really short finish to Yesnaby the second day. If you don't fany the walk, there is a bus that will take you to the start of the Yesnaby road, it's a short walk from there to the cliffs, bus info here: https://www.orkney.gov.uk/Service-Directory/B/Bus-Services.htm
I think there is a water tap at Rackwick but I'm not 100% sure. Probably trickier for Yesnaby and North Gaulton because most of Orkney Mainland is farm land and I'm not sure I'd want to drink out of the streams there unless you filtered the water first.
Edit: There is a lot of convenience to be gained by having a car on Orkney Mainland for tourism and going to the supermarket, etc. It is worth considering what you will do if it craps it down with rain for a couple of days - with a car you can get to the indoor wall or swimming pool or tourist sights etc. much more easily.
Post edited at 11:40