Consider adding a by() function, to split up large it() statements

[nikita-leonov]

There was a question before opened that are similar to my one (#138). I think it makes sense to implement beforeAll that executed at beginning of example group but not each time when new nested It called. Such approach supported by Kiwi, Sleipnir, RSpec, Cucumber etc.

Here is a simple scenario that tested not efficiently with beforeEach instead of beforeAll

describe("Friends list") {
  context("for a newly registered firstUser", {
    beforeAll({
      //Create new user on backend
    })
    it("is empty.", { //Check })
    context("after secondUser registered", {
      beforeAll({
        //Create new user on backend
      })
      it("is empty.", { //Check })
      context("after firstUser sent friends request to secondUser", {
        beforeAll({
          //Create friends request
        })
        it("is populated.", { //Check })
      })
    })
  })
}

With beforeEach instead of beforeAll it will be 6 networks calls instead of 3 if I am calculating it right.

Is there any conceptual explanation why beforeAll is not implemented available in a framework?

[modocache]

Is there any conceptual explanation why beforeAll is not implemented available in a framework?

Yep! The short answer is: beforeAll and afterAll encourage shared state among examples, so I've expressly chosen to not support them.

For a detailed discussion, check out onsi/ginkgo#70, wherein I ask the exact same question you're asking now! 😄

If you feel strongly about it, feel free to comment on this issue. For now, though, I'm closing this. Thanks for the suggestion, though! 👍

[nikita-leonov]

I expected an answer like this. I agree with "pureness" of a test stage, it is nice that we have beforeEach sequences that allows us even in a very deep example groups get a untouched environment for test. However, there are cases where it is not working very well. I have a nested test with 3 levels of context nesting and multiple Its on each level, as result instead of ~ 6 network queries I have close to 50.

I believe XCTests are running one by one, so parallelization argument is not applicable for Quick, while decision about using beforeEach vs beforeAll and pureness of test environment should be up to developer of tests.

Pretty often time of tests execution is critical for development process and such methods like beforeAll / afterAll should help a lot in this.

ps I just remembered that mocha (http://mochajs.org) also have before & after that allows to define one point of initialization.

[modocache]

Awesome arguments, @nikita-leonov! You make some very good points that were nice for me to think about as well. Let me see if I can put in my two cents on each of them...

However, there are cases where it is not working very well.

This is definitely true. For example, you may want to limit the number of times expensive operations are run. beforeAll may help, but for these cases, I would use mocks in place of those expensive dependencies. If that's not possible, there's also beforeSuite.

I have a nested test with 3 levels of context nesting and multiple Its on each level, as result instead of ~ 6 network queries I have close to 50.

Using HTTP stubbing libraries like Nocilla or OHHTTPStubs, I've rarely found network requests to be prohibitively expensive in tests. When they do become a burden, I take it as a sign that my networking client contains too much logic, and I separate its responsibilities out into another class.

I believe XCTests are running one by one, so parallelization argument is not applicable for Quick

True, XCTest runs are serial. But as mentioned in the discussion in onsi/ginkgo#70, not supporting beforeAll/afterAll also makes Quick simpler to maintain. For example, like Ginkgo, Quick will also support test order randomization soon; see #152. Implementing this feature will probably be much simpler thanks to not needing to support beforeAll/afterAll.

Pretty often time of tests execution is critical for development process and such methods like beforeAll/afterAll should help a lot in this.

I agree that tests should be fast, but I don't think beforeAll/afterAll help as much as you imply. For truly expensive operations, I would advise using mocks, not sharing state.

So I'm afraid I'm still not in favor of this issue. But I'm open to suggestions. @Quick/contributors, any thoughts?

[jeffh]

I agree with @modocache. Just stub the network requests, it's an external dependency that's probably slowing down the unit tests.

If you're writing integration tests, that's expected to be slower and having repeat requests is okay. Alternatively, you can coalesce your it blocks together - since its should really only contain assertions. But then again, integration tests are more like smoke tests. They are to verify the interaction with the external dependency is correct and not cover all the edge cases.

beforeAll is a significantly complicated feature that taxes other possible features for a feature that's arguably not useful for a unit testing framework. Talking to the Jasmine folks, there are a lot of hidden complexities that doesn't make it seem worth it. Most features of Jasmine had to be reviewed to for the interaction with beforeAll/afterAll was desired (spy cleanup, nested beforeEach & beforeAll execution, making sure execution order is valid).

It's much easier write your own non-generialized beforeAll than it is for Quick to support this feature.

[nikita-leonov]

Gents, thanks for such a comprehensive clarification of thoughts regarding Quick implementation. As of now my personal opinion is also on your side in terms of not implementing beforeAll / afterAll. I was trying to hit two birds with one stone by implementing full stack acceptance testing with a backend involved while still be performance efficient in terms of amount of requests. I think for such type of tests a small overhead of requests caused by nesting should not be a problem. If such problem exists then it should be a different type of test where the testing environment is mocked but not real.

I was thinking about another scenario, that kind of off-topic, but what about UI testing with use of Quick? I used Kiwi before for UI testing and it was very helpful to run multiple tests on a prepared screen, but not go through the complex UI flow again and again for each it separately. Solution of bundling multiple its in one I am treating as bad practice and do not review in scope of this questions.
Disclaimer: As my currently ongoing tasks are not related to UI testing I am not really pushing for beforeAll and afterAll for handling of UI testing scenario, but just want to know you opinion how the testing with Quick should be done in this particular case, as we may do UI testing someday at future.

Thanks!

[jeffh]

I think that question would be great to ask on stackoverflow. Just email me the link and when I get time to sit down and answer I will. 

[evgenyneu]

It would be cool to have beforeAll/afterAll functions. We split our integration test into small it functions for readability sake. It would be nice to have beforeAll do the database setup before all those it are executed. We use same concept in Rails as well with before(:all).

[modocache]

I'm still against adding beforeAll and afterAll, but noticed I made a few statements in this discussion that I now disagree with.

Using HTTP stubbing libraries like Nocilla or OHHTTPStubs, I've rarely found network requests to be prohibitively expensive in tests.

I've since started advocating for an adapter pattern which introduces a protocol for the networking layer. For details see @jeffh's excellent talk, Testing in Swift.

True, XCTest runs are serial. But as mentioned in the discussion in onsi/ginkgo#70, not supporting beforeAll/afterAll also makes Quick simpler to maintain.

To expand, I think open-source projects like this one invite feature bloat. In addition, keeping maintainers active and knowledgeable about the system is much harder than it sounds. Adding complexity is not something to be taken lightly!

[piofusco]

I have been using Quick/Nimble along with Apple's UI Testing framework to build out an acceptance test suite (we point our app to a mock server which starts/stops before the test suite is run). Keep in mind, this is true UI testing in that there is no means to "reach into the app and muck about."

In order to keep tests easier to read, most are it blocks with single assertions - this make failures easier to read as well. Grouping them together in a single it just doesn't really make sense.

Being that I am without beforeAll or afterAll, this requires me to start my XCUIApplication before each test. As you can imagine, this is quite costly.

While I appreciate the purity of not sharing state across tests, I am not sure it's applicable to this case. Wouldn't it be grand if I could start up my app once for any group of tests?

App Starts
Test 1
Test 2
Test 3
App Stops

I understand writing acceptance test is something that is very uncommon for mobile projects due to lack of reliability and support (particularly from Apple). I won't get preachy, but, in combination with unit tests, they provide immense value.

I also appreciate the concern for feature bloat - to end, here are my questions:

• Is Quick only to be used for unit testing? If so, why?
• If there is no desire to add beforeAll, is there a means to define multiple suites within a single UI Testing target? If each had a beforeSuite it would/could be functionally be equivalent to a before(:all).

[modocache]

Is Quick only to be used for unit testing? If so, why?

No, ideally we should be able to do anything XCTest is built to do, but with a different DSL.

I can appreciate the fact that there are prohibitively expensive operations, such as starting an XCUIApplication, that make beforeAll useful. It seems like demand for this feature has increased since XCUITest was released, which jumbled Apple's unit testing framework together with its UI testing framework. It also seems like those circumstances mean we're dealing with different problems than Ginkgo, whose approach I have referenced in this thread.

So, I've changed my mind: maybe it is time to add beforeAll and afterAll to Quick. If anyone's interested in contributing this functionality, please copy the other functional tests in this repository to make sure it all works. I think this is a tricky feature, and I'd hate for it to become a maintenance burden. It'd also be great if whoever contributed it came to join Quick's group of committers. :)

[modocache]

Also, not that he works on this project, but I'd be super curious to hear what @onsi has to say on this topic.

[onsi]

:) I've written up my thoughts on this previously for Ginkgo:
onsi/ginkgo#70

As y'all can see there have been several follow on issues asking for this. I continue to contend that the feature is unnecessary and limiting as it couples individual specs in ways that prevent things like parallelization and randomization by explicitly introducing a connascence of order to the test suite. Almost always the concern is around performance. Two things here: first class parallelization support (made possible by not having BeforeAlls) is one way to address performance. Second, when that isn't practical or doesn't produce enough performance gains - I personally prefer to solve this problem by having larger Its and using Ginkgo's By command to annotate the It and break it up into sections (this is actually considered a best practice in Agouti the web-driver acceptance framework built on top of Ginkgo). I appreciate that isn't always possible or preferable.

At the end of the day it's also generally straightforward for a developer to ensure the contents of a BeforeEach only run once so if you really want this you can do it yourself.

But I'm being a purist ;) @modocache if your customers are bringing reasons unique to the iOS ecosystem that can't be resolved by these other approaches then the pragmatic thing may be to add it as a first class feature.

[modocache]

it couples individual specs in ways that prevent things like parallelization and randomization by explicitly introducing a connascence of order to the test suite

Oh wow, I had completely forgotten about this. We do have open issues tracking randomness (#152) and parallelization (#383).

I personally prefer to solve this problem by having larger Its and using Ginkgo's By command to annotate the It and break it up into sections

I hadn't heard of By -- I'll try looking into it and seeing if it fits here as well.

But I'm being a purist ;) @modocache if your customers are bringing reasons unique to the iOS ecosystem that can't be resolved by these other approaches then the pragmatic thing may be to add it as a first class feature.

I'm struggling with whether this is the case. In the end, it's all about test suite performance, so there's nothing inherently different here. On the other hand, Quick is built on top of a system that is not parallelizable, so adding first-class support to alleviate the problem isn't an easy option. #383 tracks adding parallelization, but we have a big yak to shave: decoupling Quick from XCTest. That's a lot of work!

I think the pragmatic thing to do here is:

• Evaluate whether including a By construct works as a middle-of-the-road solution.
• If By is no good:
  • Add beforeAll and afterAll. This prevents us from randomizing the test order in the short-/medium-term, which stinks. Then again, work on Provide option to randomize test order #152 hasn't actually started, so we're just blocking a new feature, not removing an existing one.
  • In the long term, and as part of a major release, remove beforeAll and afterAll, and in their place provide tools to parallelize.

[modocache]

Thanks @onsi! Your input is really valuable to me!! ❤️

[nikita-leonov]

@piofusco usually you do not expect acceptance tests to be fast. It may take days for big application to verify all UI states etc. In such long runs reliability of test results is more important than speed anyway. I believe that sharing a state contributes into reliability significantly. I assume that even with an introduction of beforeAll it is a question of time when you get critical mass of tests to make your test suite to be relatively slow again. So introducing of beforeAll will temporarily mask a fact that acceptance tests are slow, but will not be able to solve you problem.

[piofusco]

usually you do not expect acceptance tests to be fast. It may take days for big application to verify all UI states etc. In such long runs reliability of test results is more important than speed anyway.

Yup - I completely agree, @nikita-leonov, 👍 . However with Apple's new UI Testing framework, I disagree with

I believe that sharing a state contributes into reliability significantly ... introducing of beforeAll will temporarily mask a fact that acceptance tests are slow, but will not be able to solve you problem.

The problem right now is time - not state pollution. Other than using gestures, there isn't really any state to be mucked around with.

Currently, we have many it blocks with single assertions within a single spec() for individual elements. Having to start and stop the XCUIApplication before each is very expensive - beforeAll would solve this as I would be able to

Start up the application
Test 1
Test 2
Test 3
Stop the Application

Having the ability to group tests being made on the same view with a single application startup would cut down on time tremendously.

Sure - acceptance tests aren't fast, but a beforeAll would make things way faster than the way things are currently working

Start up the application
Test 1
Stop the Application
Start up the application
Test 2
Stop the Application
Start up the application
Test 3
Stop the Application

Or perhaps I'm misunderstanding you?

[rudolf-adamkovic]

Fascinating discussion!

[jeffh]

Hey @piofusco,

I still think what I said earlier still holds true:

It's much easier write your own non-generialized beforeAll than it is for Quick to support this feature.

It's easier to write your own that doesn't have to answer a myriad of questions that Quick needs to resolve. Here's some random examples:

---

beforeEach { printf("beforeEach") }
context("something") {
   beforeAll { print("beforeAll") }
   it("") { ... }
}

Does beforeAll print before or after beforeEach?

---

beforeAll { printf("1") }
context("something") {
   beforeAll { print("2") }
   it("a") { ... }
   context("else") {
      beforeAll { print("3") }
      it("b") { }

      context("blah") {
          beforeAll { print("4") }
          it("c") { }
      }
   }
   beforeAll { print("5") }
}
beforeAll { print("6") }

What is the order of prints for numbers assuming the its execute in order?

What is the order of prints for numbers assuming the its execute in reverse order?

What's the output if the tests execute in it("c"), it("a"), it("b")?

Are all those forms of execution equivalent?

---

sharedExamples("a shared example") { (sharedExampleContext: SharedExampleContext) in
   beforeAll { print("inside shared example") }
   it("") { }
}

beforeAll { print("beforeAll 1") }
itBehavesLike("a shared example") { [...] }
context("something") {
   beforeAll { print("beforeAll 2") }
   itBehavesLike("a shared example") { [...] }
}

What is the order of the lines printed assuming its execute in order?

What is the order of the lines printed assuming its execute in reverse order?

Is it identical execution for these behaviors?

---

There are many different permutations that can be made with many of Quick's features that require similar questions. While some may have more obvious executions than others, some have no obvious "correct" way of executing. And we haven't thought through afterAll yet.

[piofusco]

@jeffh - however it works in Rspec/Specta/etc. No need to reinvent the wheel.

I won't pretend to understand the technical difficulties in implementing this feature, but I stand by my argument that it would be incredibly useful for writing UI tests because of how expensive starting/stopping the simulator can be (especially when running CI). I'll still keep on using this library and wishing it had beforeAll.

As someone who uses Quick/Nimble every day, I appreciate this library and the work that goes into it immensely! Please don't misunderstand from my curt response.

From everything above (your post and other members), it's clear the shoe doesn't really fit for writing UI Tests (yet). As such, I've started writing my UI tests without Quick, but have kept using Nimble for the matchers because of the changes brought on with new versions of Xcode/AppCode.

[jeffh]

Just to be clear @piofusco, I didn't intend to sound offended, my probably comes off more rough without emojis 😅.

Considering Rspec & Specta have differing behaviors, I'm not sure it's safe to say there's an obvious reference implementation. Rspec controls test execution, so I don't think that implementation available to us to copy (at the moment). And Specta has somewhat surprising behavior (like how afterAll runs after the first afterEach executed, not after all tests in that given context finishes).

The most amendable implementation is the one @onsi used in ginkgo (using By), but I'm not sure how much value that adds (maybe enough?).