This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionLast revisionBoth sides next revision | ||
test:review-system [2008/11/03 20:24] – fantasai | test:review-system [2010/08/02 14:40] – fantasai | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== | + | ====== |
See also [[http:// | See also [[http:// | ||
- | * Check off X tests and give them all the same comment | + | ===== Expected Workflow ===== |
- | * Find test by ID, URL, title, etc. | + | |
- | * Handle renaming of test | + | |
- | * Track changes in same stream as comments (every subversion checkin adds a comment pointing to diff and new version) | + | |
- | * Track status of test (needs review, needs work, etc) | + | |
- | * Accept comments | + | |
- | * Be test suite aware (which test belongs to which test suite); note that a test may belong to multiple test suites | + | |
- | * Opt-in to hear comments on tests you reviewed (and accepted but second-level review found problems with?) | + | |
- | * Easy to set up an account | + | |
- | * Bulk submit preserving folder structure | + | |
- | * Should work on all browsers | + | |
- | ====== Workflow ====== | + | ==== Workflow for Test Author ==== |
- | + | ||
- | ===== Workflow for Test Author | + | |
- Author identifies an assertion to test | - Author identifies an assertion to test | ||
Line 31: | Line 19: | ||
- Author receives acknowledgement that tests have been accepted and checked in | - Author receives acknowledgement that tests have been accepted and checked in | ||
- | ===== Workflow for Reviewer | + | ==== Workflow for Reviewer ==== |
- If the Reviewer has ' | - If the Reviewer has ' | ||
Line 41: | Line 29: | ||
- Otherwise, Reviewer enters comments explaining what changes need to be made and sets the status to ' | - Otherwise, Reviewer enters comments explaining what changes need to be made and sets the status to ' | ||
- | ===== Workflow for Approver | + | ==== Workflow for Approver ==== |
- Approver searches the submission database for tests in the ' | - Approver searches the submission database for tests in the ' | ||
Line 47: | Line 35: | ||
- Approver may suggest a new filename for checkin. | - Approver may suggest a new filename for checkin. | ||
- | ===== Workflow for CVS Monkey | + | ==== Workflow for CVS Monkey ==== |
- If the test is now in the ' | - If the test is now in the ' | ||
+ | |||
+ | ===== Feature Requirements ===== | ||
+ | |||
+ | * Check off X tests and give them all the same comment | ||
+ | * Find test by ID, URL, title, etc. | ||
+ | * Handle renaming of test | ||
+ | * Track changes in same stream as comments (every subversion checkin adds a comment pointing to diff and new version) | ||
+ | * Track status of test (needs review, needs work, etc) | ||
+ | * Accept comments | ||
+ | * Be test suite aware (which test belongs to which test suite); note that a test may belong to multiple test suites | ||
+ | * Opt-in to hear comments on tests you reviewed (and accepted but second-level review found problems with?) | ||
+ | * Easy to set up an account | ||
+ | * Bulk submit preserving folder structure | ||
+ | * Should work on all browsers | ||
+ | * Unified login with wiki and Subversion | ||
+ | |||
+ | ===== Backend Design Notes ===== | ||
+ | |||
+ | A major design principle is to integrate with Subversion so that changes in SVN get automatically reported to the review system. Another is to take advantage of the tests' metadata so that as much information as possible will be automatically pulled from the test source. | ||
+ | |||
+ | ==== Database Tables ==== | ||
+ | |||
+ | |||
+ | |||
+ | test suite | ||
+ | suite short name | ||
+ | suite full name | ||
+ | url prefix | ||
+ | | ||
+ | tests | ||
+ | id # | ||
+ | testID (filename without extension) - should be unique (error if not) | ||
+ | path in svn | ||
+ | metadata (from test file) | ||
+ | author(s) | ||
+ | help link(s) | ||
+ | title string (unique) | ||
+ | assertion | ||
+ | flags tokens | ||
+ | status | ||
+ | checked-in | ||
+ | current revision? | ||
+ | owner (username) initial value is from initial checkin | ||
+ | | ||
+ | comments | ||
+ | id (of test) | ||
+ | date | ||
+ | comment | ||
+ | commenter (username) | ||
+ | validation script result (enum) | ||
+ | revision at time of comment | ||
+ | status change | ||
+ | commenter name (if anon user) | ||
+ | | ||
+ | svn status (tracking what happened in svn) | ||
+ | id | ||
+ | committer svn username | ||
+ | rev | ||
+ | date | ||
+ | comment | ||
+ | commit type (update, | ||
+ | | ||
+ | - need to track svn renames | ||
+ | | ||
+ | | ||
+ | user (drupal) | ||
+ | username | ||
+ | |||
+ | realname | ||
+ | role (svn access) | ||
+ | |||
+ | ==== UI Pages ==== | ||
+ | |||
+ | Eira's mockups: http:// | ||
+ | |||
+ | === full report for one test === | ||
+ | |||
+ | * meta data presented at top | ||
+ | * merged list of comments / changes (link to diff) sorted by date | ||
+ | * logged in user may change status and comment | ||
+ | * anonymous user may make a comment, filling out either a name field or filling out username+pass to log in (like on LiveJournal) | ||
+ | |||
+ | === generic query interface === | ||
+ | |||
+ | * list by status | ||
+ | * search on assertion, title, testid, author, help link, who commented | ||
+ | |||
+ | === query results/ | ||
+ | |||
+ | * batch review, list tests, link to test, checkbox, review | ||
+ | * list title, assertion, link to test, link to full report | ||
+ | * only peers can approve | ||
+ | * flag tests with revisions since last comment | ||
+ | |||
+ | === interface to rename/move test files === | ||
+ | |||
+ | * Should allow substring matching to rename/move multiple files | ||
+ | |||
+ | ==== URL Scheme ==== | ||
+ | |||
+ | * review/ | ||
+ | * review/ | ||
+ | * review/ | ||
+ | * review/ | ||
+ | * (query based on test suite) | ||
+ | * review/ | ||
+ | |||
+ | ==== For Later ==== | ||
+ | |||
+ | * generate email on comment or commit, submitter, reviewer (all commenters since last status change) | ||
+ | * profile page to edit user data & email prefs | ||
+ | * handling of tests in checked-in state: In this system, | ||
+ | * Approved state requires checked-in. | ||
+ | * Switching status to Approved automatically moves test to approved dir and marks as checked in | ||
+ | * And so if you search Approved status, it searches approved directory | ||
+ | * You can mark an Approved test as NeedsWork or Wrong: this svn copies it into the submitted directory and tracks it there. It keeps its checked-in status so we know this is high priority as it's live. Once re-approved, | ||
+ | * Tests never lose their checked-in status unless they' | ||