Back to Venmo.Com

QuizTrain: A Swift framework for TestRail's API

· by David Gallagher

QuizTrain: A Swift framework for TestRail's API

At Venmo we love using TestRail to manage and track tests for our mobile and web apps. We wanted a way to integrate it with our automated iOS tests which are written in Swift. TestRail provides an excellent API but no Swift framework to use it. So we decided to build one and open source it under the MIT License so you can use it too!

QuizTrain allows you to do everything you can do with TestRail's API from the comforts of Swift on iOS, macOS, tvOS and watchOS. Acting as a foundation you can integrate it with your Swift testing framework or even build an app on top of it. We've been using it at Venmo since early 2018 and absolutely love it. 😀

How Venmo uses QuizTrain

On iOS we use Apple's User Interface Testing framework to run automation tests. It provides a great way to test Venmo from the perspective of users while remaining within Apple's development ecosystem. TestRail acts as our single source of truth for all test cases and results. Keeping this in mind we leveraged QuizTrain to do the following:

  • As tests run we log rich test case descriptions pulled from TestRail.
  • We track results as pass/fail based on assertion outcomes in tests.
  • When complete we create a test plan and submit results to TestRail.

This was accomplished by building a few objects on top of QuizTrain and extending XCTContext. You can find an example project in the QuizTrain repository containing this code.

  • TestManager.swift
    • Principal Class for our testing bundles.
    • This sets up QuizTrainManager before any tests run.
    • Contains global methods to log, start, and complete testing case IDs.
  • QuizTrainManager.swift
    • Conforms to and is registered for XCTestObservation.
    • Tracks cases being tested.
    • Tracks failed assertions.
    • Marks cases as passed/failed.
    • Handles submitting results to TestRail.
  • QuizTrainProject.swift
    • Stores a copy of our TestRail project in memory.
    • Provides rich descriptions for cases.
  • XCTContextExtensions.swift
    • Extends XCTContext.runActivity() to accept test case IDs to log, start, and complete tests.

The resulting test code is very simple:

class MyTestCase: XCTestCase {
    func testSomething() {
        XCTContext.runActivity(testing: 14539) { activity in
            XCTAssertTrue(true, "It was not true!")
        }
    }
}

That's it! What's happening:

  • testSomething() starts.
  • An activity is created for case ID 14539.
  • A rich description is logged for it.
  • Testing is started for the test case.
  • The activity closure is executed making assertions and completes.
  • Testing is completed for the test case.

If any assertions fail between steps 4 and 6 then the test case ID is marked as failed. Otherwise it passes. Later when results are submitted the failure message is appended to the result description so you can view it on TestRail. If multiple failures occur between steps 4 and 6 then every failure is included in the result description.

Multiple test case IDs can be grouped together and/or nested:

class MyTestCase: XCTestCase {

    func testSomethingElse() {

        XCTContext.runActivity(testing: [13803, 14002, 14016]) { activity in
            XCTAssertFalse(true, "It was not false!")
        }

        XCTContext.runActivity(testing: 13293) { activity in
            XCTAssertTrue(true, "If this fails then 13293 fails.")
            XCTContext.runActivity(testing: [13830, 13044]) { activity in
                XCTAssertTrue(false, "If this fails then 13293, 13830, and 13044 fail.")
            }
        }
    }
}

One rule we implemented is you can only start and complete a test case ID once during the entire test run. Otherwise a fatalError() will occur. Our reasoning for this is:

  • It enforces that our test cases are succinct.
    • If a test case is used more than once it is a sign that we made it too vague and we should break it up into multiple test cases.
  • It is DRY and prevents us from testing the same thing more than once.
  • It avoids the problem of how to deal with conflicting results for a test case.
    • What if it passed, failed, passed, and also had a custom result? How would you merge that into a single result?

You're more than welcome to modify QuizTrainManager.startTesting() if you prefer different behavior.

Have Fun!

We hope you enjoy using QuizTrain with TestRail as much as we have! Check out the QuizTrain readme and example project if you'd like to learn more. Happy testing! 📝🚆