16:00:05 <jlaska> #startmeeting
16:00:22 <jlaska> #meetingtopic Fedora QA Meeting
16:00:33 <jlaska> #topic gathering bodies
16:00:52 <jlaska> I think we have a few folks out today ... adamw and f13
16:01:18 <jlaska> show of limbs for folks hanging out for the QA meeting today
16:01:31 * Southern_Gentlem 
16:01:34 * Viking-Ice ..
16:01:40 <jlaska> Southern_Gentlem: Viking-Ice: greetings!
16:01:41 * skvidal is here
16:01:46 <jlaska> skvidal: howdy :)
16:02:32 * wwoods appears
16:03:06 <jlaska> wwoods: hey hey!
16:03:23 <jlaska> alright ... let's get started
16:03:47 <jlaska> I've got a proposed agenda out to the list, so if no objections, we'll just walk the agenda
16:04:12 <jlaska> agenda - https://www.redhat.com/archives/fedora-test-list/2009-June/msg00652.html
16:04:20 <jlaska> #topic Previous meeting follow-up
16:04:38 <jlaska> a short list of follow-up items from last week ... and I'm fail on the first item
16:04:42 <jlaska> * [jlaska] - update [[QA/Goals]] wiki document
16:04:44 <jlaska> :(
16:04:57 <jlaska> Still high up on my list ... just trying to figure out a good way to represent ideas and let it grow
16:05:28 <jlaska> I'm referencing the community architecture goals page as a starting point
16:05:36 <skvidal> jlaska: interpretive dance works well in the wiki
16:05:44 <jlaska> skvidal: no kidding! :)
16:06:08 <jlaska> so this will remain on my list for next week
16:06:41 <jlaska> we've got a lot of cool things going on ... and a some common themes to the work, I just want to wrap some wording around it
16:06:48 * jlaska struggling with that atm
16:07:10 <jlaska> okay, the only other item on the list is sort of vague ... but I know work has been happening there
16:07:14 <jlaska> * [wwoods] - updates on Rawhide acceptance test plan
16:08:39 <jlaska> did we lose wwoods?
16:09:01 <wwoods> sorry
16:09:08 <jlaska> oh good, no worries
16:09:17 <wwoods> so we have a test plan: https://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan
16:09:31 <wwoods> it suggests 10 test cases for rawhide acceptance
16:09:48 <wwoods> we filled out 10 tickets in the autoqa trac to write those test cases
16:09:59 <wwoods> (see the link in the Test Cases section)
16:10:05 <jlaska> got some good feedback on the list for the plan
16:10:09 <wwoods> I've written one already: https://fedoraproject.org/wiki/QA:Comps_Validity_Test_Case
16:10:19 <jlaska> thanks to ldimaggi_ for pitching in!
16:10:38 <wwoods> yeah, the list of test cases etc. was determined after some discussion on fedora-test-list and on the wiki
16:11:26 <jlaska> wwoods: using the format you specified in the wiki text, I added my nick to the list of reviewers
16:11:28 <wwoods> anyone who wants to help write up (or review) the test cases.. we'd appreciate it
16:11:51 * jlaska encourages others who want to help review the test plan (or listed test cases) to do the same
16:13:21 <jlaska> wwoods: nice, any other updates on that front, anything to carry forward to next week?
16:13:28 <Viking-Ice> Arent we forgetting in the "Basic Functionality" to check if an network connection can be established ?
16:14:34 <wwoods> hm. yeah, that's worth checking
16:14:39 <Viking-Ice> no shit :)
16:14:42 <wwoods> as is seeing if yum can pull down packages
16:14:47 <wwoods> (which sort of implies network functionality)
16:14:48 <Viking-Ice> exactly
16:15:01 <Viking-Ice> the whole being able to update :)
16:15:19 <wwoods> okay, good call
16:15:47 <jlaska> I'm not quite comfortable yet on where to draw the line between acceptance testing of rawhide ... and more functional testing
16:16:09 <skvidal> the line in my mind is this
16:16:37 <skvidal> acceptance testing is 'do all the bits fit somewhere' is it POSSIBLE to install this given the depsolving requirements
16:16:43 <skvidal> AND
16:16:54 <skvidal> can a scripted installed actually succeed
16:17:06 <skvidal> if those two both test out correctly then, rawhide is acceptable
16:17:07 * jlaska lines that up with the comments in the introduction
16:17:09 <skvidal> beyond that is functional testing
16:17:42 <wwoods> Well, a scripted install as we typically run them
16:17:47 <wwoods> would require network and yum to work
16:17:54 <wwoods> since anaconda would be using those to fetch and install packages
16:17:56 <wwoods> right?
16:18:24 <jlaska> wwoods: I was wondering that as well
16:19:07 <skvidal> wwoods: well, it requires the LOADER network works
16:19:16 <skvidal> it doesn't require that the booted kernel network works
16:20:49 <jlaska> okay, so the take away is to add an additional network check on the post-installed system?
16:21:16 <wwoods> it's trivial to test, since typically our tests are going to report their status using the network
16:21:17 <skvidal> <shrug>
16:21:22 <jlaska> right
16:21:27 <skvidal> wwoods: fair enough
16:21:32 <wwoods> but there's thousands of different ways it can possibly fail
16:21:38 <wwoods> so it's not a very, uh, focused test
16:22:04 <wwoods> but then, irb.com is answering a very general question
16:22:27 <skvidal> international rugby board?
16:22:29 <wwoods> I dunno. I can go either way here. Technically we can perform installs without network
16:22:43 * skvidal must have missed something
16:22:53 <jlaska> #action add post-install network test to basic functionality in rawhide acceptance plan
16:22:56 <wwoods> but the vast majority of our tests (and testers) would be kinda busted if network or yum stopped working
16:23:06 <wwoods> skvidal: sorry - shorthand for israwhidebroken.com
16:23:22 <skvidal> oh right, sorry
16:23:32 <skvidal> I actually went to irb.com
16:23:48 <skvidal> curious how international rugby board and is rawhide broken have a lot of things in common, though
16:24:01 <wwoods> Being able to test yum functionality is the highest priority for the critical-path stuff
16:24:07 <jlaska> good point
16:24:17 <wwoods> so that proposal (and ensuing test plans/cases) will definitely cover that use case
16:24:52 <wwoods> stuff left off the Rawhide Acceptance Test Plan will *not* be ignored
16:25:07 <wwoods> but we won't even bother looking at the critical path tests until the acceptance test passes
16:25:21 <skvidal> agreed
16:25:37 <Viking-Ice> kinda goes without saying..
16:25:41 <wwoods> so the informal line is something like this: can someone actually test this thing. "Is Rawhide Broken?"
16:26:02 <wwoods> and I think most of us would consider it broken if networking was completely non-functional
16:26:09 <wwoods> or yum was unable to upgrade any packages
16:26:30 <jlaska> okay, we have a few topics to cover today ... any objections to moving on?
16:26:31 <wwoods> failures of *individual* network chipsets do not count as "broken"
16:26:33 <wwoods> nor wireless.
16:26:39 <jlaska> are there any other things we want to capture here?
16:26:41 <Viking-Ice> agreed
16:26:51 <wwoods> in short: yeah, Viking-Ice is right - we need test cases for network and yum
16:27:05 <wwoods> very, very basic test cases
16:27:18 <jlaska> does it smell ... [yes] [no]
16:27:24 * poelcat wonders if any of the tests or open tasks could be queued up for working on at the FUDCon in Berlin this weekend?
16:27:27 <skvidal> so you can do both, actually
16:27:37 <jlaska> poelcat: we can talk about that shortly
16:27:38 <skvidal> at the same time
16:27:46 <jlaska> poelcat: I've got an update from f13 on that subject
16:27:54 <wwoods> skvidal: right - the implementation of the test cases will be pretty trivial
16:28:06 <jlaska> #topic Fedora 12 QA schedule changes
16:28:17 * poelcat just saw http://spevack.livejournal.com/84441.html
16:28:47 <jlaska> poelcat sent an updated F12 QA schedule to the mailing list last week
16:28:57 <jlaska> See https://www.redhat.com/archives/fedora-test-list/2009-June/msg00507.html
16:29:27 <jlaska> I wanted to talk about it briefly here to see what, if any, concerns the QA folks had about the schedule?
16:29:52 <jlaska> to hilight some changes that I keen on ...
16:30:17 <jlaska> A few of the informal activities that happened during F11 are now a bit more formal
16:30:28 <jlaska> like the scheduling of blocker bug days
16:30:49 <jlaska> and compose+hand_off of release candidate trees to QA
16:31:07 <jlaska> But poelcat has asked for additional guidance from the QA group
16:32:10 <jlaska> if I think back around F11, those 2 items listed above were items where it was unclear who was doing what when ... so I dig those changes
16:32:54 <jlaska> poelcat: are there any other areas of the schedule you're looking for feedback on?
16:33:17 * jlaska still on the fence with getting test days in the schedule
16:34:47 <jlaska> Viking-Ice: wwoods: do you folks have any concerns about the proposed changes?
16:35:03 <jlaska> or perhaps, looking for something extra?
16:36:00 * jlaska takes silence as violent agreement :)
16:36:13 * poelcat also wanted to volunteer to create an ical file for QA tasks if there is interest... something similar to http://poelstra.fedorapeople.org/schedules/f-12/f-12-releng.ics
16:36:16 <wwoods> are there more frequent bug reviews in this schedule than in the F11 schedule?
16:36:39 * poelcat stepped away and is back
16:36:42 <jlaska> wwoods: yeah ... there are now 3 bug reviews ... 1 before freeze, and 2 following (prior to RC composes)
16:36:45 <poelcat> wwoods: yes
16:36:51 * Viking-Ice stands with jlaska on being on the fence with getting the test day's in the schedule..
16:37:10 <jlaska> Viking-Ice: I'm there mostly since that schedule is *so* dynamic still :(
16:37:30 <wwoods> yeah, that was the only thing other than test-day scheduling that I wanted to be sure of
16:37:46 <jlaska> okay cool
16:37:56 * poelcat thinks wiki is probably best for the test days unless they will always happen on certain days
16:38:28 <jlaska> they always happen on Thursdays ... but that can change based on need
16:38:33 <poelcat> the other big distinction was two specific composes for a release and then subsequent testing
16:38:33 <Viking-Ice> I dont really want to nailed the test days to down to a schedule  we need the flexibility
16:38:36 <jlaska> or at least are planned for thursdays
16:38:48 <jlaska> Viking-Ice: okay
16:39:19 <poelcat> so next step would be to see if releng can accomodate composes on the days proposed
16:39:42 <poelcat> and maybe plan for them in advance?
16:40:42 <jlaska> sounds good to me ... that gives us some time to prep the test results reporting and any other QA call for actions
16:40:45 <poelcat> i was try to formalize some of the past discussion about limited QA capacity and not being able to test every compose
16:41:12 <jlaska> assuming rel-eng is okay with them, I think it's worth a try for the upcoming release
16:41:58 <jlaska> poelcat: okay ... any other points you wanted to raise?
16:42:09 <poelcat> that's it
16:42:49 <jlaska> oh, I forgot your iCal idea
16:43:36 <jlaska> I don't have any strong feelings on that yet
16:43:37 * Viking-Ice brb..
16:44:00 * jlaska is used to going to the wiki 12/Schedule page
16:44:15 <jlaska> might be neat to try for this next cycle
16:44:24 <jlaska> but I can't say whether or not it's critical
16:45:09 <jlaska> if anyone else has input or thoughts on providing an ical schedule, please do follow-up with poelcat
16:45:15 <jlaska> #topic AutoQA - update from wwoods
16:45:43 <jlaska> wwoods: we may have tackled a fair bit already, but I have a spot in the agenda for an update on the autoqa project
16:46:00 <wwoods> well, the step after writing up a bunch of test cases is.. automating 'em
16:46:24 <wwoods> I think we've got some docs on setting up autotest
16:46:41 <jlaska> yeah I think I put the link in the ticket ... 'cause I knew I'd forget
16:46:42 <jlaska> ...
16:46:58 <jlaska> https://fedorahosted.org/autoqa/ticket/5
16:47:20 <wwoods> so Real Soon Now we'll be trying to write some test code and wikifying what we learn about writing tests for autotest
16:47:20 <jlaska> the KVM guys did most of the hard work already for us ... http://www.linux-kvm.org/page/KVM-Autotest/Server_Install
16:48:50 <wwoods> so I expect next meeting (or the one after that) we may have some test code and/or results to show
16:49:01 * Viking-Ice back
16:49:06 <wwoods> and a list of cases we need help writing
16:49:31 <jlaska> wwoods: when you say writing, do you mean automating or just defining wiki test cases?
16:49:48 <wwoods> sorry, automating
16:49:48 <Viking-Ice> probably both :)
16:50:03 <wwoods> writing test cases in the wiki has already started and will be ongoing
16:50:13 <wwoods> hopefully finishing in the next week
16:50:35 <wwoods> automating those cases can start whenever we've got time to look at it
16:51:03 <jlaska> wwoods: f13 asked earlier today on #fedora-qa what things he could ask others to help with.  I pointed to the list of test case tickets in autoqa trac
16:53:55 <jlaska> wwoods: would you like help from others on defining the test cases ... or is that a head down solo activity?
16:54:47 <wwoods> help from anyone who understands how we test this stuff already would be much appreciated
16:55:03 <wwoods> we're really just writing down all the unwritten requirements
16:55:30 <jlaska> would it make sense to send something out to the list asking for interested volunteers?
16:55:57 <jlaska> I mean, if we think this is an activity folks might be interested in contributing
16:56:02 <wwoods> once we have that written up properly, anyone with a little python skill could probably figure out the autotest API and write tests
16:56:10 <wwoods> sure, I'll do that
16:56:37 <jlaska> maybe we setup some time on irc for folks to gather and discuss/share ... dunno?
16:58:15 <jlaska> f13 also passed along a URL for a presentation he is giving @ FUDCon this week around autoQA (http://jkeating.fedorapeople.org/presentations/automatedqa.odp)
16:58:33 <jlaska> wwoods: any other updates on the autoQA front?
16:59:44 <wwoods> nothing I can think of offhand
16:59:48 <jlaska> okay
16:59:53 <jlaska> #topic   Test Day SOP draft - update from awilliam
17:00:11 <jlaska> adamw is out today, but just a quick update on some documentation posted to the list for review
17:00:31 <jlaska> Adam's words do a better job explaining it than mine ... https://www.redhat.com/archives/fedora-test-list/2009-June/msg00662.html
17:00:43 <jlaska> summary: a draft SOP around hosting a Test Day
17:00:53 <jlaska> Any feedback or thoughts to the list are welcome!
17:01:38 <jlaska> okay ... last agenda item on the list ...
17:01:42 <jlaska> #topic Improve Bug Reporting proposal - update from viking_ice
17:01:59 <Viking-Ice> https://fedoraproject.org/wiki/User:Johannbg/QA/Improve_reporting
17:02:06 <Viking-Ice> Read that :)
17:02:15 <jlaska> You posted a link to a draft proposal last week.  I wanted to give you a chance to let folks know where things are with the proposal
17:02:22 <jlaska> and what sort of feedback you were looking for
17:03:17 <Viking-Ice> First status updates on QA task that I defined are on the draft
17:03:51 <Viking-Ice> There are good news for reporters that have thought bugzilla reporting form is to complicated
17:05:15 <Viking-Ice> when our bugzilla is upgraded to 3.4 ( which will happen not so long after it's release ) it will contain a simpler form by default ( user can click advanced to bring more elements into the reporting form )
17:05:27 <Viking-Ice> see --> http://lpsolit.wordpress.com/2009/05/24/simpler-form-to-file-bugs-in-bugzilla-3-4/
17:06:19 <wwoods> oh cool
17:06:23 <jlaska> Ah very nice ... so does that solve the first item you've listed in the proposal?
17:06:35 <Viking-Ice> yup
17:06:36 <jlaska> re: "Bugzilla reporting UI too complicated."
17:06:40 <wwoods> interestingly, it mentions http://hendrix.mozilla.org/
17:06:46 <wwoods> which I hadn't heard of before
17:07:21 <Viking-Ice> Now this is the big deal on the draft "Lack of needed information for maintainer(s) to be able to successfully work with the report."
17:09:04 <jlaska> Viking-Ice: you've got a lot of QA tasks listed in the proposal ... are these things you will be doing, or are looking for help on?
17:09:40 <Viking-Ice> would not mind help
17:10:35 <Viking-Ice> Perhaps abrt has gather all the needed info for us?
17:10:45 <jlaska> have you raised your ideas during bugZapper meetings?
17:10:56 <Viking-Ice> nope
17:11:29 <Viking-Ice> usually dont make appearance on bugzapper meeting
17:12:08 <jlaska> I ask because some of the areas of improvement line up with what the BugZappers team is trying to accomplish
17:12:20 <jlaska> perhaps a good way to team up?
17:12:27 <Viking-Ice> Yeah sure
17:13:28 <Viking-Ice> The main thing we need to solve is getting the info from maintainers on what to report store that in a db and reguire reporter to attache those files when reporting
17:13:54 <Viking-Ice> does any one know how is abrt solving this issue ?
17:14:03 <jlaska> would it make sense for a more focused document on that topic?  I know if working together during previous test days, this is a topic near & dear to ya
17:14:51 <jlaska> Viking-Ice: wwoods and I were discussing that yesterday.  The project is active, but I'm not in the loop on their upcoming milestones
17:15:34 <Viking-Ice> Ok
17:15:49 <Viking-Ice> I'll try to digg up some info how they are addressing this issue
17:15:58 <jlaska> okay
17:16:17 <wwoods> IIRC they had (or were planning to have) plugins or conf files that specify what files to attach
17:16:23 <Viking-Ice> now  "Duplicated reports" already filed upstream improvements to the new simplified reporting form
17:16:24 <wwoods> there's some notes on the abrt wiki
17:16:32 <wwoods> (https://fedorahosted.org/abrt/wiki)
17:17:04 <Viking-Ice> to atleast reduce somewhat the possibility on that
17:17:54 <Viking-Ice> https://bug499320.bugzilla.mozilla.org/attachment.cgi?id=384113
17:18:18 <jlaska> Viking-Ice: were there any other items you wanted to address during the meeting today?
17:19:08 <Viking-Ice> not really
17:19:33 <jlaska> anything you want to come back to for next weeks meeting?
17:19:48 <Viking-Ice> Anyway we need to come up with some plans on how to gather the info from maintainers on what they want on their reports on how to get that info from them
17:19:55 <Viking-Ice> and store it etc..
17:20:25 <Viking-Ice> so both bugzilla and possible abtr can use it
17:20:34 <jlaska> I think that is something concrete enough that could generate good future discussion
17:21:21 <Viking-Ice> If someone feels something lacking feel free to express it on that page
17:21:30 <Viking-Ice> I've also started on https://fedoraproject.org/wiki/User:Johannbg/QA/Kernel
17:21:52 <jlaska> nice
17:21:56 <Viking-Ice> with regards to j-rods wish on the F11RR meeting
17:22:05 <jlaska> seeing your second page makes it a little clearer what you're trying to do
17:22:15 <jlaska> I could see that growing rather nicely
17:23:11 <Viking-Ice> These are just rough sketches ( both the improvements and the KTT )
17:23:20 <jlaska> right
17:23:23 <Viking-Ice> So I can keep track on things
17:23:47 <jlaska> for me ... like we've talked about before, just having a way to catalog content for testers to easily find would be a great start
17:24:37 <jlaska> anyhow ... let's open up for general discussion
17:24:42 <jlaska> #topic Open discussion
17:24:42 <Viking-Ice> ok
17:25:02 <jlaska> we've gone way over today ... but I missed time for open-discussion last week
17:25:08 <jlaska> any topics/questions/concerns folks want to raise
17:25:50 * jlaska sets the fuse to 2 minutes
17:26:31 <jlaska> ... 30 seconds ...
17:26:33 <Viking-Ice> this may come as a stupid question but who are the target audience with israwhidebroken.com and what is being hope to accomplish with that ?
17:26:53 <jlaska> Viking-Ice: ah ... not a stupid question
17:26:56 <Viking-Ice> we testers ( atleast most of us  ) know rawhide is more or less broken up to beta
17:27:29 <jlaska> yeah, I think we all know, but don't have concrete data readily available to prove it
17:27:30 <Viking-Ice> I read the FAD report but failed to see the purpose of this
17:28:10 <jlaska> pulling from the proposal ... "Provide a single, well-known location with information about whether or not Rawhide is broken, and a link to the last known-good Rawhide tree. "
17:28:20 <Viking-Ice> which servers who
17:28:35 <Viking-Ice> serves
17:28:48 <jlaska> all of this leads up to providing a known good rawhide ... which serves QA for bug verification, feature testing, test days
17:29:13 <jlaska> and the release team, by providing a history on the health of rawhide as we approach milestones (or are past them)
17:29:32 <Viking-Ice> Ah ok
17:29:39 <jlaska> for me ... it's all about consistently gathering data to facilitate decision making
17:30:21 <Viking-Ice> yup..
17:30:33 <jlaska> does that help answer your q?
17:30:40 <Viking-Ice> indeed it does
17:30:43 <jlaska> cool!
17:31:02 <jlaska> alrighty ... unless any other thoughts, let's wrap it up
17:31:30 <jlaska> #endmeeting