How to take payments as a startup?

I wouldn't spend too much time trying to go against the norm here – Stripe is the default option. Whether you're using subscriptions, payment links, or direct debit; Stripe's integrated services are the best choice. With features like automatic invoicing, Stripe Checkout, tax products for when you grow, and a payment CRM, Stripe has everything you need.

That being said, Stripe has become less customer-friendly and is now an incumbent, and as a result, they are beginning to add increasingly more charges—for example, VAT to invoices. For accounting context, they have a direct bank feed to Xero (which is great for revenue recognition) but not with QuickBooks. 

Adyen only really works with enterprises and large customers. We dropped them a line, and their sales team told us that we were too small to be interesting. GoCardless & Stripe won't connect directly to QuickBooks, so you may end up paying £ 100 for tools like Synder, which you can connect to your accounting software to get any reporting granularity. These tools are really not very good. Even if Stripe is becoming less customer-friendly over time, it is quite a safe bet, ideally combined with Xero instead of QuickBooks.