I'm using flutter too, I've found a website called codemagic, so I can make builds for iOS. My laptop is windows. The bulk of development is building for Android.
It's nearly impossible to get away from Xcode entirely. You will always need the simulator and probably some entitlement / asset tools. What you can do is write the bulk of your app as an SPM package using your editor of choice, and then include that as a local or GitHub repo dependency in your Xcode shell app (which has no code, just assets, preview assets and plist files)
I use Xcode, but just to run the app with Command-R so I can see log messages or stack traces if I need to drop it into the agent for analysis. Everything else is done by Claude Code / Codex.
You could probably use Emacs for Swift...just like for about every other language.
Whether you still must download a monstrosity is left as an exercise for the reader.
[0] https://codeberg.org/luxmentis/xcede
I even read the source using Zed not Xcode :)
That would make it an web app tho. Not sure if that's what OP wants when making Swift apps.