fastlane 3.0 - plans, ideas, discussions

[joshdholtz]

👋 Hey, fastlane community!

It's been a while since I've posted a discussion 😅 Been real busy with family, personal stuff, and work. I try to keep my fastlane time as productive as possible moving PRs forward and closing issues so sadly I can't spend as much time in discussions as I'd like. However... this topic is important and it deserves its time and input from all of you!

fastlane 2.0

It's been almost 5.5 years since the fastlane 2.0 release. This release was a big one for fastlane as it combined all of the tools into the one fastlane gem. It was a major change for both fastlane users and
maintainers. We have tried to provide back support and avoid as many breaking changes as possible since then and I'd like to say that we've done pretty well at 207 minor versions later 😇 But fastlane 3.0 has been on my mind for some time now. There are some things I'd like to change that would make fastlane even more user friendly, stable, maintainable, and resilient going forward.

fastlane 3.0

I don't have any concrete plans in mind for fastlane 3.0 yet. I do, however, have a few general areas that I would like to improve:

• Spaceship
• Bridging tools and actions together
  • Specifically migrating away from tool codenames
• Action organization
• Improved web and command line docs
• Namespaced lanes in Fastfile

Spaceship

This one is big and the recent App Store Connect API 2.0 release is really sparked me to write this discussion 🙃

History of Spaceship

Feel free to skip this section as its kind of long

Spaceship was originally built to interface with the Apple's private APIs for Apple Developer Portal and iTunes Connect (now App Store Connect). The only form of auth was using Apple ID and having a custom written module for this was the best way for fastlane to programmatically automate. The number of APIs grew quickly but also broke quickly has these were undocumented APIs that were never guaranteed to be there or behave the same.

In 2018 at WWDC, Apple announced the App Store Connect API. Not relevant but I was there for the announcement and it was awesome 😊 It was everything we wanted to replace the our usage of Apple's private APIs. Having an official API that was well documented, stable, and built for exactly what we were doing was perfect. The hard part is that we were using A LOT of APIs and it was going to take a while for the official App Store Connect API to grow to where we needed it to be.

We slowly started to integrate what we could with the official App Store Connect API in 2019, even more so in 2020, and now most of our major tools and actions can fully use this official API as of 2021. The hard part is that now every fastlane user can use the App Store Connect API keys which are required to use the App Store Connect API. The user that cannot use this are developer that don't have permission to create the keys (only account admins can) and enterprise users.

Spaceship got a big overhaul to support both the API Key and Apple ID auth versions of App Store Connect API in 2020 and 2021. Because of this, we have not been able to automatically generate code based off of the OpenAPI spec that Apple provides for the App Store Connect API. All of the endpoints up to this point have had to been manually written. There also was no OpenAPI spec right away so our initial design decisions could not be based off of that anyway 🤷‍♂️

Future of Spaceship

App Store Connect API 2.0 released over 104 new endpoints and most of them were added for in-app purchases and subscriptions. I was going through the diff between 1.8 and 2.0 and realized that there is no way that we would be able to implement those endpoints as fast as we needed to. It got me thinking that fastlane needs to start considering a rework of Spaceship.

I want fastlane 3.0 to start using code generated from the App Store Connect API OpenAPI spec. The benefits here would be huge for the fastlane users. They would get access to new API's almost right away and everything would have documentation. There is also a huge benefit to the fastlane maintainers once this is designed out for similar reasons. This is HUGE undertaking though because we still need to support Apple ID auth for some users. This would need to be designed in a way where its easy enough for fastlane's maintainers and users to still have the same Ruby interface for either authentication type even though they are hitting different hosts.

I think that is that for now. This would be a big design task that would need to be planned, coordinated, and implemented perfectly to provide as little change for users. But there will be changes and probably breaking and that is why it would feel better to go into fastlane 3.0.

Bridging tools and actions together

The tools and actions living in the same environment was introduced in fastlane 2.0 (which I mentioned about above a little bit). fasltane 1.0 pulled in all of the tools as separate gems. These tools were things like deliver, gym, snapshot, scan, and a few more. Actions lived as part of the fastlane gem. They are essentially very focused methods that do one thing. This could be performing git functions, uploading IPAs to test services, or modifying Xcode project plists and build settings.

There was clearly a difference between these in 1.0 and at the start of 2.0. But as 2.0 grew, the number of actions grew and the usage amount of actions grew. Tools and actions are built completely differently inside of fastlane but they behave the exact same when using a Fastfile. The only main different with tools is that they offer their own command line interface (like fastlane gym or fastlane deliver) since they were originally their own tools.

The tools have names that aren't very descriptive to what they do (ie: gym, scan, snapshot, match, sigh, cert, pem). This is because the tools lived other own before being merged into fastlane. In fastlane 2.0, the tools got aliases to what they did (ie: gym became build_app, scan became run_tests, snapshot became capture_ios_screenshots). Calling tools by the code names made sense for early fastlane users because that is what they were but new fastlane users will most likely refer to their description names. This can get confusing when tools build_app will have references to gym without any reasoning.

Future of tools and actions

Tools and actions need to almost become the same in fastlane 3.0. The code names should be removed to provide more clarity to new users. Tool and actions should also no longer be two separate things. Actions should gain some structure and grouping like the tools have. The tools have a special CLI implementation that some actions could also benefit from.

This idea is not super well planned out and I would love to get feedback from maintainers and users on this one to see if the pains and confusions and need for changes matches my thoughts.

Action organization

This is kind of tied to the bridging of tools and actions. Currently, all of fastlane action are thrown into one single directory. It's messy. It doesn't provide good organization or code sharing of actions that should be grouped together (ex: git actions, Xcode project actions).

We have been encouraging the creation of plugins for any new functionality to lesson the load of fastlane maintainers and to make new functionality available without the release of a new fastlane version. However, there are still a lot of things that are core to many fastlane users that should be actions.

It would be great to have a new organization and structure of fastlane's action in 3.0 to make things more manageable to maintain which would provide more stable and powerful actions going forward.

Improved web and command line docs

This one doesn't necessarily need to go into fastlane 3.0 since its not breaking but web and CLI docs have not seen many improvements over the past few years. This could be because the docs were already so great or it could be because there were so many other things to do 😛 But... if we will potentially be reorganizing tools, actions, and providing more App Store Connect API endpoint integrations, we might as well rethink and rework the docs 💪

Goal of this

This ended up being A LOT longer than I originally planned 😇 I don't know if I just felt like writing or I just wanted to give some history of fastlane but here we are. I want to push fastlane forward to 3.0 making it an even better experience for both our users and our maintainers

What I'm looking for

• Thoughts, questions, and general feedback on the 3.0 breaking changes I've mentioned above
• Any thoughts or ideas you all my have for fastlane 3.0
• Anybody looking to help contribute to this future effort
• Anybody brave enough to test pre-releases of 3.0

Thank you everybody for reading this and thank you to those who will be contributing thoughts!
I love fastlane and I love this community ❤️

[AliSoftware]

Actions should gain some structure and grouping like the tools have

💯 on better structure and grouping, not only in the code, but also at usage site.

Especially the growing number of actions, I would love related actions to gain the ability to get grouped into some kind of namespace, to make them more discoverable and organized, but also make the calls to those actions from a Fastfile more clear what they relate to.

---

Related, could I shamelessly suggest we include #18164 in fastlane 3.0 as well? In other words, if we bring some kind of concept of grouping/namespacing actions to make them more organized, I'd also love this ability to be provided to lanes that users write in their Fastfile as well.

In our case, to give you an idea, we have so many lanes that we decided to split them into separate fastlane/lanes/*.rb files that we then import from the main Fastfile, to better organize the code of related lanes together… that's saying something! But sadly that only helps in organizing our Fastfile code, but doesn't namespace the lane names themselves…

[joshdholtz]

Yes! I will update with the description with namespaced lanes! That would be 🔥

[KrauseFx]

Maybe there is a way to organise actions like people organise their emails: previously we have used folders, but nowadays we use tags we add to emails, to make it easy to find & filter. Could we do something similar for fastlane actions, e.g. take_screenshots being #screenshots #simulators #marketing and alike?

[AliSoftware]

That's an interesting idea 🤔 but I don't think the only problem here is finding the right action (as in fastlane actions --filter-by-tag screenshots), but is also to make it more readable in the Fastfile what the action is about (and potential what context it's about)

For example we could imagine having git::add and git::commit and git::tag_exists, but also have <plugin-name>::<action>.

—

One typical example we have on our end is that we have a release-toolkit fastlane plugin that contains a lot of useful actions to help us in our release process. But because of lack of namespacing, we end up having a hard time finding proper names for all those. There are some actions that make sense only for iOS, others only for Android, others work for both, multiple actions are about localization wrangling (sending / downloading strings from our translation platform/service), other actions are about interacting with our CI (triggering builds, etc) etc. Having namespacing embedded in the name of those actions would help make our Fastfiles clearer when calling those actions (and we could imagine have some autocomplete support in IDEs for Fastfile that take advantage of this namespacing in the future, even), in a way that tags like you propose won't 🙃

All that being said… why-not-both.heif

[CraigSiemens]

+1 for some kind of grouping/namespacing.

pilot/testflight is an example of an action that could benefit from namespacing. The CLI interface has some subcommands like pilot upload, pilot add, and pilot find but they aren't all available through the pilot tool in a fastfile. Having namespaces could allow using the same subcommands in the fastfile something like testflight::testers::add or testflight::build::upload. It also tries to so a lot of things that could be separated like, upload, wait for processing, submit for review, add groups to build, etc.

Some other benefits it could add:

• Consistency with the command line interface.
  • fastlane testflight builds would be the same as testflight::builds in the fastfile.
• Group related actions together that are currently separated
  • latest_testflight_build_number could become testflight::build::latest_build_number or just testflight::build::latest if it returned both the version and build numbers.
• Simplify the documentation for the parameters. Many of the tools have lots of parameters but not all of them make sense to be used a the same time.
  • groups is a parameter that only needs to be used when other parameters are set like distribute_external. It takes some scrolling around in the docs to find out which ones need to be used together.

---

Part of me keeps wanting a more OO approach where you get instances and call methods on them.

build = testflight::build::upload
build.wait_for_processing(timeout: 60)
build.add_group("External Testers")

But it feels like it could add complexity keeping a similar interface available though the CLI.

[mollyIV]

Huge +1 for "Bridging tools and actions together" and "Action organization" ❤️

The current number of actions can be somewhat overwhelming, especially without robust discoverability support. Even to this day, I often struggle to locate a specific action without having to double-check the documentation 😉

Inconsistencies in naming, especially when different names refer to the same tool, can make it harder for users to develop lanes effectively.

I am pretty confident that improving naming conventions, enhancing the overall structure, and introducing better namespace organization will significantly enhance the user experience.

[rogerluan]

This is all amazing! Thanks for kicking off this discussion 💪 I'll start different threads for each topic/idea that I'd like to bring up, so they can be discussed in their own thread 😁

Looking forward to hearing everyone's thoughts! 🤗

[rogerluan]

1. Deprecating everything in the codebase that is marked to be deprecated 😄 it's time! (and maybe also look for other things that deserve to be deprecated and just wasn't properly tagged/commented)

[rogerluan]

2. This one is a bit trickier, I'd like to gauge interest/thoughts: having very few "macro scopes" (or Gemfile groups) we could use to separate dependencies. For instance: I don't need anything related to Firebase/Google Play in my local bundle if this is a pure iOS repo - this causes problems sometimes, and waste space/slow things down for dependencies we're not using.

[rogerluan]

One caveat here is that it does make maintenance harder AFAICT. It would be a trade off between "keeping people's environments cleaner/faster" VS "maintainability cost" perhaps. Hopefully if we can think of a way that reduces the maintenance cost, it'd become a more interesting option to evaluate

[rogerluan]

3. Similar to the previous one (2), we could separate match from its 3 storage types (s3, git, google cloud) - these could be plugins that implement a common protocol/interface, and this would also allow for users to implement their own storage methods perhaps.

[vadimbelyaev]

I'd love to see more documentation with example on how to use Fastlane with GitHub actions.

Also it would be great to improve Fastlane's ability to build an app without using match for code signing. I personally find match a bit too intrusive. As an alternative, I'll be just happy to feed Fastlane the provisioning profile and the .p12 file as base64 encoded strings from secrets. But I wasn't able to find any good documentation or examples of that.

P.S. Josh, a huge thanks for the awesome work you and other maintainers are doing.

[rogerluan]

@vadimbelyaev that's great feedback, but it's not related to fastlane 3.0 IMO :)

• Documentation on fastlane + GitHub Actions: https://docs.fastlane.tools/best-practices/continuous-integration/github/ - if you miss anything, or you know how to improve it, feel free to open Issues / PRs at https://github.com/fastlane/docs
• Build without match: you can use other actions like https://docs.fastlane.tools/actions/update_code_signing_settings right before invoking fastlane's build action. There are other combos, but you'd need to find the right candidate for your project specifically. If you feel like the documentation could be better, a PR is always welcome 🤗 (again, at https://github.com/fastlane/docs 😉)

[rogerluan]

4. I think we should also upgrade the minimum Ruby version supported, from 2.6 to either 2.7 or 3.0 already 😬 There's an added cost to keep maintaining old Ruby versions, e.g. downgrade danger to keep supporting Ruby 2.6 docs#1183

[rogerluan]

5. Perhaps move actions that offer a too-specific solution to a problem, to be a community-managed plugin. Every now and then I find actions that I ask myself "who uses this? this is too specific." — we can probably reduce the overhead of maintaining those actions by moving them to plugins instead. The idea is that should they ever require maintenance, users would open issues in the community plugin repos instead of in this main repo, and the community repos can be maintained by more people (the community) with reduced risks.

This would be pretty much the same rationale as the one described here https://github.com/fastlane/fastlane/blob/master/VISION.md#actions-and-plugins except we'd be moving "hardly used" actions to become plugins (back-filling our current philosophy).

Separate question which would help us: do we have analytics on how frequent our actions are used/called? Maybe if we can list them all based on frequency of use, we can move the ones that are barely used to become plugins instead.

[ainame]

Another idea that can break compatibility with fastlane 2.x but would love to have to avoid awkwardness. This is relevant to restricting input value types.

Currently, given parameters, regardless of how to pass, go through the auto-conversion, which means to exist for environment variables and CLI. This can result in some unfortunate when it comes to inline action calls in Fastfile. For example, this ticket is what I had #20745

fastlane/fastlane_core/lib/fastlane_core/configuration/configuration.rb

Lines 80 to 89 in 950bb82

	def verify_value_exists
	# Make sure the given value keys exist
	@values.each do |key, value|
	next if key == :trace # special treatment
	option = self.verify_options_key!(key)
	@values[key] = option.auto_convert_value(value)
	UI.deprecated("Using deprecated option: '--#{key}' (#{option.deprecated})") if option.deprecated
	option.verify!(@values[key]) # Call the verify block for it too
	end
	end

To mitigate the issue, I think we should ideally skip the auto-conversion in parameters/options when we don't need it, like when you pass inputs explicitly from Fastfile in Ruby.

# bad (this is allowed in fastlane 2.x)
some_action(
  option_that_expects_array: "1, 2, 3"
)

# good
some_action(
  option_that_expects_array: [1, 2, 3]
)

I think we can introduce deprecation messages for this behaviour in fastlane 2.x and then drop it completely when hitting the major bump.

[joeyErlandson]

Better Fastlane.swift documentation and support!

I would imagine similar to my team, other teams and individuals would prefer to work with Swift as opposed to Ruby since most/all of our other work is in Swift.

[mollyIV]

Contributions

While it might not be directly tied to potential changes in fastlane 3.0 (although it could be), I want to emphasize the significance of simplifying the contribution process. This involves:

• Streamlining the setup of the development environment, perhaps through tools like GitHub Workspaces or VSCode Containers.
• Improving the ease of testing, especially for challenging-to-test fastlane tools/actions, such as spaceship. We could explore the possibility of providing a testing server that emulates App Store Connect responses.
• Enhancing the accessibility of getting started instructions. Currently, newcomers may need to dive deep into the contributing documentation to understand how to initiate changes in fastlane's codebase.

Thanks 😊

[mollyIV]

Enhancing Console Log Output 📝

Fastlane's console log output may be due for an update. In my opinion, it currently feels a bit outdated, and often generates more noise than necessary. For instance:

$ bundle exec fastlane foo
[✔] 🚀 
+----------------------------------+---------+----------------------------------+
|                                 Used plugins                                  |
+----------------------------------+---------+----------------------------------+
| Plugin                           | Version | Action                           |
+----------------------------------+---------+----------------------------------+
| fastlane-plugin-secret_keeper    | 1.0.0   | add_item_to_keychain,            |
|                                  |         | read_item_from_keychain,         |
|                                  |         | remove_item_from_keychain        |
| fastlane-plugin-versioning       | 0.5.2   | ci_build_number,                 |
|                                  |         | get_app_store_version_number,    |
|                                  |         | get_build_number_from_plist,     |
|                                  |         | get_build_number_from_xcodeproj  |
|                                  |         | , get_info_plist_path,           |
|                                  |         | get_version_number_from_git_bra  |
|                                  |         | nch,                             |
|                                  |         | get_version_number_from_plist,   |
|                                  |         | get_version_number_from_xcodepr  |
|                                  |         | oj,                              |
|                                  |         | increment_build_number_in_plist  |
|                                  |         | ,                                |
|                                  |         | increment_build_number_in_xcode  |
|                                  |         | proj,                            |
|                                  |         | increment_version_number_in_pli  |
|                                  |         | st,                              |
|                                  |         | increment_version_number_in_xco  |
|                                  |         | deproj                           |
+----------------------------------+---------+----------------------------------+

[08:45:53]: ---------------------------
[08:45:53]: --- Step: shell command ---
[08:45:53]: ---------------------------
[08:45:53]: ----------------------------------------
[08:45:53]: --- Step: Verifying fastlane version ---
[08:45:53]: ----------------------------------------
[08:45:53]: Your fastlane version 2.213.0 matches the minimum requirement of 2.116.0  ✅
[08:45:53]: ------------------------------
[08:45:53]: --- Step: default_platform ---
[08:45:53]: ------------------------------
Fastfile:17: warning: already initialized constant Fastlane::FastFile::CHANGELOG_FOLDER_ABSOLUTE_PATH
NotificationsFastfile:3: warning: previous definition of CHANGELOG_FOLDER_ABSOLUTE_PATH was here
[08:45:53]: Driving the lane 'ios ci_post_code_review_metrics' 🚀
[08:45:53]: -------------------
[08:45:53]: --- Step: is_ci ---
[08:45:53]: -------------------
[08:45:53]: --------------------------
[08:45:53]: --- Step: ensure_is_ci ---
[08:45:53]: --------------------------
+------------------+---------------------------------+
|                    Lane Context                    |
+------------------+---------------------------------+
| DEFAULT_PLATFORM | ios                             |
| PLATFORM_NAME    | ios                             |
| LANE_NAME        | ios foo |
+------------------+---------------------------------+
[08:45:53]: 🚫  This lane can be executed only by CI

We have the opportunity to streamline and improve the logs, making them more readable, user-friendly, and informative. To find inspiration, we can look at how tools like npm, yarn, or tuist handle their logs.

I'd love to hear your thoughts on this! 🙇

[andreasgangso]

Consistency in parameter naming.
E.g. some places xcodeproj is used, some places path.
Some places code_signing_identity other places codesigning_identity.

[rogerluan]

I'm always all-in for consistency!

[rogerluan]

Brainstorming: how would/should we find those cases that need fixing? Can you think of an automated way to do so? If not, how could we be as exhaustive as humanly possible?

[andreasgangso]

Don't know if there are any simple solutions to this. Might need to skim through every action and list up everything that seems off.

Maybe open an issue where people can comment every time they find some naming that annoys them.
I actually searched for something along the lines of "inconsistency" and "naming" in issues before I posted here.

[andreasgangso]

Another thing that would decrease the need for this is some docs on getting code completion when writing lanes in popular editors, e.g. vscode.
I'm currently freehanding them, but if I had IDE hints i wouldnt have cared about inconsistencies and would never have posted this.