watchOS support

[MarcoSero]

Quick (and Nimble) need to add a watchOS target to support the new watchOS architecture.

[modocache]

Awesome!! You probably know more about watchOS than I do--care to submit a pull request?

[MarcoSero]

I got started on this but there I'm already facing the first issue. There seems to be no XCTest framework for the watchOS simulator.
The folder /Applications/Xcode-beta.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/Library/ does not contain Frameworks/XCTest.framework. A bit weird.
Also, when creating a new target there is not watchOS Test section in the list of templates.

Did Apple say anything about running tests on the watch during WWDC?

[phatblat]

Yes. Xcode 7 supports unit and UI testing of watchOS 2 apps. Creating new watchOS app includes a Unit Test and UI Tests target by default.

The .xctest bundle for the watchOS tests are built in the Debug-iphonesimulator folder, so I take it that they link against the iphonesimulator version of XCTest.framework.

[MarcoSero]

@phatblat Yes, that option does exist but those tests are for the normal app target.

[MarcoSero]

This should probably be kept open until Apple fixes rdar://21760513, but until then there's not much we can do.

[karapigeon]

Noticed that the Radar is still open but has there been any developments with this?

[modocache]

I know very little about watchOS. From what I understand, it doesn't really have XCTest...? So we may indeed by blocked by Apple on this one. If someone here has actually developed for the Apple Watch, their insight would be great to have on this issue.

[jeffh]

FWIW, cedar did support watchOS to some degree. I haven't checked recently if it still works. Most of the heavy lifting was done in PivotalCoreKit which faked out the classes being inherited. I'm not sure if that has been updated to newer watchOS SDKs.

[karapigeon]

Sounds good to me. On one hand, I think supporting watchOS would be good but on another, if Apple hasn't supported it internally in Xcode with a test target, I don't see the point because the tests would be in the normal watch target (if I'm understanding correctly).

[karapigeon]

We could close this for now and reopen it in the future. There's hope for watchOS 4, haha.

[jeffh]

A normal watch target isn't necessarily a bad thing. Back in the old days before test bundles were allowed on iOS, an iOS app was the target type for those writing test suites. App-target tests are easier to maintain from a technical side, but they will probably cause a larger support burden because they differ to test bundles in specific ways.

That being said, I think it's fine deferring it for now. It's not an often requested feature so I don't think it's high on the list of priorities.

[karapigeon]

I mean, if we'd like to come back to this in the future, I don't mind but I think it's fair as you said to defer it for now. If there's a random spike in popularity in the request then we can totally do it!

[karapigeon]

I'd like to close inactive or deferred issues. All of the blocked ones that we may come back to have the blocked tag on there for a reason. We can always come back to them and reopen them again if they become relevant. With that said, I'm going to close this one but we can totally reopen again if necessary.

[ikesyo]

https://developer.apple.com/documentation/xcode-release-notes/xcode-12_5-release-notes

Xcode now supports XCTest unit and UI tests for watchOS apps. When creating a new watchOS app, check the Include tests checkbox to add a unit and UI test target to the newly created project. For an existing project, add a unit or UI test target via File > New > Target, and then add the test target to the Test action of the WatchKit App scheme. To run the tests, select a watch simulator or device from the run destinations menu in the toolbar, and then choose Product > Test. Note that testing is supported on watchOS 7.4 or later. (21395998)

[danielsaidi]

Any news on this? The first error I get is that BadInstructionException isn't defined for watchOS, in Nimble's ThrowAssertion. Perhaps more errors will surface once this is fixed?

Would it be possible to just let these unsupported parts resolve to something that is disabled or stubbed out in watchOS, perhaps with an assertion error so that unit tests always fail on watchOS?

That would let us build all projects that rely on Quick and Nimble, but at least let the projects compile for watchOS. That would be a huge win. I love using Quick and Nimble, but this error makes supporting watchOS a lot harder in the projects where I use them.

[danielsaidi]

I just made Nimble compile on watchOS by just adding an #if switch to throwAssertion that throws a fatal error for watchOS, much like it already does for other unsupported platforms.

This means that throwAssertion will currently always fail for watchOS, but at least the code will compile and any developers that use throwAssertion will get a descriptive error and have the choice to disable these tests for watchOS.

I tried disabling ThrowAssertionTests for watchOS as well, and the Nimble unit tests run perfectly on watchOS.

This is the new implementation:

public func throwAssertion<Out>() -> Predicate<Out> {
    return Predicate { actualExpression in
    #if os(watchOS)     <-- NEW if check
        fatalError("Nimble currently doesn't support watchOS.")
    #elseif (arch(x86_64) || arch(arm64)) && (canImport(Darwin) || canImport(Glibc))
        let message = ExpectationMessage.expectedTo("throw an assertion")
        var actualError: Error?
        let caughtException: BadInstructionException? = catchBadInstruction {
            #if os(tvOS)
                if !NimbleEnvironment.activeInstance.suppressTVOSAssertionWarning {
                    print()
                    print("[Nimble Warning]: If you're getting stuck on a debugger breakpoint for a " +
                        "fatal error while using throwAssertion(), please disable 'Debug Executable' " +
                        "in your scheme. Go to 'Edit Scheme > Test > Info' and uncheck " +
                        "'Debug Executable'. If you've already done that, suppress this warning " +
                        "by setting `NimbleEnvironment.activeInstance.suppressTVOSAssertionWarning = true`. " +
                        "This is required because the standard methods of catching assertions " +
                        "(mach APIs) are unavailable for tvOS. Instead, the same mechanism the " +
                        "debugger uses is the fallback method for tvOS."
                    )
                    print()
                    NimbleEnvironment.activeInstance.suppressTVOSAssertionWarning = true
                }
            #endif
            do {
                _ = try actualExpression.evaluate()
            } catch {
                actualError = error
            }
        }

        if let actualError = actualError {
            return PredicateResult(
                bool: false,
                message: message.appended(message: "; threw error instead <\(actualError)>")
            )
        } else {
            return PredicateResult(bool: caughtException != nil, message: message)
        }
    #else
        let message = """
            The throwAssertion Nimble matcher can only run on x86_64 and arm64 platforms.
            You can silence this error by placing the test case inside an #if arch(x86_64) || arch(arm64) conditional \
            statement.
            """
        fatalError(message)
    #endif
    }
}

@ikesyo I will create a PR for Nimble, after which I will bump the Nimble dependency in Quick and check if the unit tests work there as well after this fix. If so, we can hopefully close this issue after creating a new Nimble version.

You can check out the pull request here.

[danielsaidi]

I can confirm that Quick compiles and the unit tests run for watchOS after linking in the fix above.

If you merge the Nimble PR and create a new patch version of Nimble, then make Quick use that version, I think we should be able to close this issue. 🤞

[danielsaidi]

@ikesyo Any progress on this? I'd love to get my PR merged and released, so that I don't have to use my own forks in my various watchOS supporting projects. Lmk if I can help in any way.