I have a Note 3 and when I have a slowdown, I back up my files ad do a clean install. Problem solved. You also have the option of rooting your phone and installing any of a number of great ROM's which significantly speed-up your phone. I'm running Lollipop 5.1 and my phone is plenty fast.

I went from a Note3 to the Note 5. The 5 works a lot better. I got the 64gb version and lack of storage hasn't been an issue, yet. The phone is smoother than the 3 and works better and faster overall. Battery life is slightly worse though. I went to the Note 5 mainly for the camera. It is a big improvement over the camera in my Note 3. I hope the Note 6 incorporates the feature set of the S7 and S7Edge. If so I will be upgrading yet again. I don't use the stylus a lot but I hate to give it up completely.

Same issue here: have the Note 3, Note 4 - not enough of an upgrade, Note 5 - no expandable memory and no replaceable battery. My phone's working just fine, but a better low-light camera, Marshmallow, micro-SD and waterproofing would make it a definite buy. A stylus is a must for me now: nothing beats it for jotting down an idea with text and sketch when you're out and about.

Samsung: here an idea for your note 6 or note 7: a waterproof note phone with magnetic stylus, that when it attached to the phone feels like being a seamless part of the phones body. Sharp edges could be solved with use of softer materials. Anyhow, that would be my dream. And oh.. don't ditch that IR transmitter, and add a dual camera with 3D scanning ability on the back... That together with an even bigger battery would be really my dream ;-)

I have the note 3 and note 4. I like the stylus on both. I also have the iPad pro stylus. It is nice and smooth, but does not have the buttons for stylus shortcuts. Unfortunately my note 4 is on the fritz, and I bought a iPhone 6S. I do miss having a stylus on it, due to my fat fingers. Hopefully they will add a Stylus in the future.

I have a 2012 Note 10.1, and it's ruined me for other tablets, lol. If a tablet doesn't have active stylus, I won't even consider it. I've had it 3 yrs, and it's still running very well. But once it goes, It'll be a Note 12.2 or a Surface Pro for my next device.
I'm not using a Note phone, but have considered it, if Samsung stops trying to follow Apple! I won't buy a phone that doesn't support SD cards, and replaceable batteries. I could maybe skip the battery thing, but no SD card is absurd. And glass backs are just an accident waiting to happen.
I'm actually hoping MS gets their act together and releases a Surface Phone soon. If they embed a stylus in the body, I would buy that in a heartbeat!

I don't use my Note 3's stylus enough. I'd be delighted to know which particular apps you find most easily used with a stylus.
I am also waiting for the rumored Note 6. Hope it's worthy of the hype about it.

njall, Squid (formerly Papyrus) is an excellent app for any Android, with active Pen support. You can purchase individual features without having to get a subscription. Subscription is one reason I don't use Evernote. When using Samsung devices, S-Notes is an excellent option. I use these 2 for the majority of my note taking. I typically use Repli-Go for PDF markups though, since it leaves it as a PDF. It doesn't have true active stylus support though, and latency isn't the best in that program. But like I said, it has it's uses. Especially if you need to sign and return something.

I want a stylus, too. But, when wishing for a phone that doesn't exist, I can't stop there. I want the sapphire screen and rugged body from the Kyocera Brigadier. I want the ability to be in charge of my own hardware from the Neo900. I want the privacy capability of Blackphone. I want the battery swapability of a marine VHF with Qi wireless charging built in to each battery. I want the externally accessible full sized SD card and stylus of a Palm Treo. I want the ability for my phone to be the communications device (Mobile Data, WiFi, Bluetooth, GPS, Glonass, NFC, RFID, InfraRed, Camera) for my laptop from the Motorola Atrix, implemented via a non-contact optical interface (if a Miele washing machine can do it...). I want Fujitsu's IR Iris scanning. And, I want the ability of LG, HTC, or Samsung to integrate it all, and sell it to me for less than $3000.