Support performance testing

[modocache]

So, my thoughts on the matter are summed up pretty well in this tweet by @luisobo. But since Quick is built atop of XCTest, it'd be great to provide fun DSL's for all of XCTest's features, including performance tests.

Not sure what the DSL should look like, though.

[bgerstle]

Calling measureBlock on an instance of XCTestCase outside a conventional XCTest method (e.g. -testFoo) results in a thrown assertion—something about numberOfTestIterationsForTestWithSelector: with a null argument returning 0 instead of 1. This is the case whether the it spec is inside a call to QuickSpec.spec or QuickConfiguration.configure.

As such, it seems that the best approach for now is to have separate XCTestCase subclasses where you put your performance tests. Would be nice to figure out what dark incantations are required to get this to hook into XCTest's strange machinery.

For kicks, I found that mixing Quick and XCTest functions in the same class doesn't work, probably due to the way QuickSpec is overriding +[XCTestCase testInvocations].

[modocache]

@bgerstle I'm not entirely certain it's related to +[QuickSpec testInvocations]--you may be describing #121. Fixing that issue may be the key to getting this working.

[skull-squadron]

On OS X (and in the iOS Simulator as a consequent, and probably on ARM too ), it's possible to use wall-time compensated rdtsc via mach_timebase_info() from xnu to get extremely precise timings. See also:

• https://github.com/bwbuchanan/absolute_time/blob/master/ext/absolute_time.c
• http://www.opensource.apple.com/source/xnu/xnu-517/osfmk/mach/mach_time.h

[karapigeon]

What's the consensus on this? Have opinions/feelings changed?

[modocache]

My thoughts have changed, as a matter of fact! I think Nimble should support performance characteristic expectations, like Ruby's MiniTest does. Something like this:

expect { foo() }.to(perform(.linearly))

[modocache]

Basically, I think it'd be cool to do this in Swift, and it'd be neat if Nimble did that. :)

Thoughts?

[karapigeon]

Oooh! That would be quite rad. I've assigned a bunch of issues to myself so at least they're assigned to someone. I'm going to try to resolve as many of them as possible and it would be cool to build something like this. Would you be able to help in the implementation side of things?

[karapigeon]

I'm all for it. Would love some help building it though. 💃

[modocache]

Sure thing! Post any questions you have here.

[Sherlouk]

😅 Nothing like a 6 year old issue to remind you how long a library has been around!

Just started exploring Quick/Nimble (unsure why I've never tried it in the past, loving it so far!) and have been exploring how best to do performance tests. It looks like this went nowhere (no worries!) but was wondering if there were any more up-to-date workarounds or extensions people are using?

In my case I'm more looking for something akin to Apple's own measure block with the ability for it to let me know if I've made a function considerably slower. I may just drop back own to a standard XCTestCase for these tests, but if there are random branches/gists etc then I'd love to trial them 🤓 Thanks