the further adventures of

Mike Pirnat

a leaf on the wind

« Previous Post

Inside the PyOhio Program Process

It has been my privilege to serve as the Program Committee Chair for PyOhio 2018 and PyOhio 2019, and I'll probably do it again for 2020. I've had a lot of positive feedback about the program we've put together these past two years, and several folks suggested that it would be worth sharing a look at my process. It seemed like a good idea at the time, so here's a peek behind the scenes of our activities and all that goes into talk selection and scheduling for our conference.

Before I begin, I want to say a couple things about PyOhio that I'm really proud of. First, PyOhio is the longest continually-running regional Python conference! We just had our twelfth year and it just keeps getting better. Second, PyOhio is the only regional Python conference that's completely free to attend, thanks to our generous sponsors. Third, all of our organizers and speakers are volunteers, to whom I am extremely grateful. Last, but certainly not least, PyOhio is proud to be a welcoming and inclusive event, a platform for a diversity of voices, and a launching pad for new speakers. It's really an honor to be leading the curation of this very special event, and I'd love to have you--yes, YOU!--make my job even more difficult next year by burying me under great talk proposals.

My first order of business is making sure we have a committee. I like to have three of us in total so that we can have different perspectives and break ties if there's contention, while also not being impossible to schedule time together (being an adult is hard, y'all).

The Program Committee's activities are tightly bound to the calendar, so it will help to start with an understanding of our timeline. PyOhio is typically held on the last weekend of July, and we aim to give speakers as close as we can to two full months to prepare their talks. I've always found it helpful to back into a deadline, so that gives us roughly:

  • End of July: conference!
  • July 1: schedule announced
  • June 8: confirmed talk lineup announced
  • June 1: first round of acceptances delivered
  • May 15: CFP closed
  • Early to mid-March: CFP opened

Around the time the CFP opens, we start pushing communications around mentorship opportunities, both checking for folks who would like some guidance on getting a proposal put together as well as those willing to mentor newer speakers. As mentors and mentees sign up, I respond first with "thanks for signing up" emails, followed by emails to mentee/mentor pairs as I group them. That's all managed via Google docs: I have Google Forms set up for mentor and mentee intake, which both dump into Google Sheets spreadsheets. I use some conditional formatting to highlight mentors who mention PyOhio in their "past history" field, on the theory that if they've attended or spoken at PyOhio, they will be more familiar with the event and the vibe we try to establish, as well as folks who have mentored speakers in the past. I add fields to these to track the dates of all the points of contact such as acceptance and pairing, and I record the mentor/mentee pairings in each sheet as well. I also track whether a mentee has a talk accepted. We ask mentees to indicate the areas they are being mentored in, like submitting a talk proposal or preparing their talks, which I use to make the pair introduction email a little more meaningful. I should probably look into automating the email processes, but the traffic has been low enough that it's been okay to just keep the email templates in a Google Doc and do the copy/paste dance as needed.

It's then pretty quiet until we approach the close of the CFP; that's when the panic sets in as we see how few proposals have been submitted up to that point. Most of our proposals come in over the final week of the CFP, with about 50% of all proposals arriving in the last day. Folks do seem to enjoy procrastination! In 2018 we closed at midnight Eastern time; we changed this year use an "anywhere on earth" (or "AoE") cutoff and that proved to be really beneficial so I think we'll probably do that going forward.

Once the CFP closes, the rush to review talks is on. Our first phase relies on input from the community, soliciting volunteers via Twitter, email, and our Slack. We try to have a meaningful number of reviews on every talk within a week of the close of the CFP. During this phase we hide the speakers' names as well as any personally-identifiable data, and we deter speakers from rating their own talks by omitting them from lists of talks to be reviewed. (Yes, this last bit relies on the honor system to a degree; please don't be a jerk.) In 2018 we changed the default sort order of proposals to be reviewed so that the first ones surfaced are those with the fewest reviews; this has been a huge improvement in making sure all the proposals get the right amount of attention and helps to even out the volume of reviews for each one. Our current review process consists of rating a proposal ("++" for "strong proposal, and I will argue for it to be included", "+" for "okay proposal, but I won't argue for it", "0" for "abstain, don't show me again", "-" for "weak proposal, but I won't argue against it", and "--" for "problematic, and I will argue against its inclusion") and mandates a comment (which for me is often the hardest part of reviewing a proposal); all of this is conveniently part of the CFP site.

This period is also critical for speaker feedback, with reviewers messaging speakers through the CFP site to ask questions, request clarifications, or suggest improvements. Ideally we'd be doing this throughout the CFP process, but with the bulk of our proposals arriving in the last 24 hours, it isn't practical until we're into the post-CFP review phase.

During this phase, the three Program Committee folks review every proposal in this anonymized form, and we each make short lists of our "must have" talks as well as anything that we think is not going to be a good fit.

The second phase starts after we have a reasonable volume of reviews for all the talks and tutorials. We disable the community access to the review process and de-anonymize all the proposals. This is often where my heart breaks as I discover that three of my "must have" talk proposals are from the same speaker and I have to decide which of my darlings to throw overboard. Optimizing for diversity of voices means that we should only be taking one talk per speaker.

I build two more Google Sheets at this point, one to track talks and the other for tutorials. Into these sheets go dumps of the title, speaker, and voting data. I also add in the voting results specifically from the Program Committe members in separate columns. Columns are added for the "verdict" (whether we will accept a talk, reject it, or hold it for backup), community vote scoring, Program Committee scoring, whether the speaker had been part of the mentorship program, and any notes we want to add. I apply conditional formatting to highlight:

  • the entire row of a 45-minute proposal
  • the "new talk" field for talks that haven't been given elsewhere
  • the "first time" fields for primary or secondary speakers who haven't spoken before
  • the "diversity statement" field for speakers who self-reported being part of an underrepresented group
  • the "mentored" field
  • the "verdict" field

Each sheet also gets a separate tab for metadata that tracks, for each type of proposal (30-minute talk, 45-minute talk, tutorial):

  • the number of proposals
  • the number of slots available
  • the number and percentage of slots filled
  • the number and percentage of slots that still need to be filled
  • the number and percentage of proposals by first time speakers
  • the number and percentage of proposals by underrepresented speakers
  • the number and percentage of accepted proposals from first time speakers
  • the number and percentage of accepted proposals from underrepresented speakers

I use conditional formatting here as well to highlight certain things as various conditions are met; this helps me to know where things are good and where we have more work to do. Also I just love cell backgrounds lighting up magically as the process unfolds. ;-)

To generate a community vote score for each proposal, I started by giving each talk 3 points for each "++", 2 points for each "+", 1 point for a "-", and no points for a "--" or abstention. Since not all proposals received the same number of votes, however, I divide the summed score by the maximum theoretically possible score (3 x number of votes) so that we end up with a percentage. I apply the same point assignments for the commitee votes, but just use the basic sum since the committee members review every talk.

From here we start the sorting churn, first to separate the 45-minute and 30-minute talks, then to rank by the community and committee scores. The first sort lets us confirm how many slots we will plan to fill. For 2018 and 2019, we ended up with forty 30-minute slots, eight 45-minute slots, and four 2-hour tutorial slots. These numbers get plugged into those metadata sheets so that we can drive all the formatting and percentages.

I make a first pass through to record all of our "must have" and "must reject" results from our short lists. Anywhere that we have either more than one "must have" from the same speaker or a "must have" and a "must reject" in conflict, I mark the affected talks with a verdict of "discuss" (with its own magic highlighting for visibility!). Then I'll work with the committee, either on a Hangouts call if we can sync up or in Slack if we need to be asynchronous, to drive out discussion on everything else, redoing the sorting grind (now including verdict as the second column, so that all the accepted things are grouped together) as we make changes. Keeping this sorting going is annoying in Google Sheets because it isn't easy to reapply a complicated multi-column sort, but it's worth it to keep things grouped together since it limits the scope of the problem and really helps us focus. This part is mostly painless until we get down into the last few slots and have to make hard decisions. It's also important to decide between tutorial and talk proposals from the same speaker; this often depends on other proposals that might do a good job of supporting, leading into, or playing off of a particular talk or tutorial.

Besides the two scores, we're also looking carefully at proposals from first-time and underrepresented speakers, as PyOhio prides itself on expanding the diversity of voices that we're giving a platform to. We continue to focus here because, while we have been very pleased with the progress we have made so far, we have a long way to go before we can be satisfied.

We also look for any underappreciated gems lurking in the lower-scoring talks; there's often something weird, different, and wonderful hiding in there that's worth boosting up.

Hopefully, all of this second phase is completed in a few days, so that by the end of the second business week after the CFP has closed, we're able to get the thumbs-up from our conference chair and send out the first round of acceptance emails. Then my true agony begins: waiting for speakers to confirm that they will actually give the talks they proposed. As confirmations come in, I add a column to my sheets next to the "verdict" for the "outcome", marking whether the talk was "confirmed", "declined", or is "pending" based on things the speaker needs to check on. This year I also added a column to the spreadsheets to record the date that the acceptance went out so that I can easily flag things that have not had a response after a couple of days. For each response that comes in, I also send a reply email to ensure a good feedback loop with the speaker; I feel it's especially important to invite folks who had to decline the acceptance to submit again in the future.

As we have speakers decline, we dip back into our maybes for replacements, checking in on Slack to discuss options, and the process repeats until we've confirmed a full talk lineup. This loop takes a speaker's geographic origin into more consideration than the first pass, since someone who is local or from an adjacent state may not have the same travel challenges that might have prevented a more distant speaker from confirming an accepted talk. This is also a step where we shine a light on talks that might have gone underappreciated during the review process. (I should note here that we have been fortunate to have some really great talks amongst the "maybe" pile that have gone on to be very well received; these aren't "bad" talks by any stretch of the imagination, they just weren't quite in the first cut.) When the lineup is fully confirmed, I'm able to tend to the unfortunate duty of sending rejection notices. When that's complete, we can announced the lineup, hopefully by about a week after the first acceptances are sent.

Once the talk lineup is confirmed, it's time to make the actual schedule! I use a site called Padlet to help with this; it's a virtual sticky note board that's simple to use and can be operated collaboratively, so it's a great fit for us. It's really nice to explore the scheduling problem spatially and quickly swap things around. I start with making an empty schedule grid, with times down the left side, and rooms, with their seating capacities, across the top. Next I make cards for all of the talks and tutorials, grouped by size/duration. Each card gets the talk ID (so that I can quickly jump to the proposal details using a custom Alfred shortcut), an abbreviated title, the speaker name(s), the speaker's state or other geographical origin point, and optionally some single-character tokens to indicate whether the speaker is a first-timer or is underrepresented. I also color-code the card if I've made any promises to a speaker about scheduling (eg, if someone has requested a particular day or time based on travel needs) and also note that on the card.

Our current conference rooms vary signficantly in seating capacity, so gauging the prospective interest in each session is vital to getting talks in appropriately-sized rooms. To do this, I create a survey in Google Forms to allow the broader community to indicate their interest in each talk. This is basically two questions with "check all that apply" answers: one for 45-minute talks and one for 30-minute talks. I enable the random ordering for the response options in each question so that we can correct for any bias in the order they're presented in. This goes out at the same time that the talk lineup is announced, and we run the survey for about two weeks, by which point we've pretty much gotten all of the responses we're going to get. I add each talk's vote total to the card in Padlet so that I can use it as a factor in room assignment.

Once all the talk cards exist, and we've got the interest data, each member of the Program Committee makes a pass through them to group them thematically, so that we can understand where we have clusters of topics or things that we want to thread together to provide an extra level of meta-narrative for attendees.

We start with scheduling the tutorials first, since there are only four of them, and they're going into a single known room. I'd like to say we use a rigorous process here, but it mainly comes down to putting them into a sequence that doesn't feel too mentally overwhelming for us if we were to attend all four (respecting any promises noted above).

Talk scheduling is a bit more challenging. I start by placing anything where we had a commitment to a speaker in roughly the right time slot, knowing I can just scoot it into a different room as I get going. In a working area, I sort the cards into columns based on the number of survey votes they received and their topical grouping. This gives me a very approximate sense of which talk is headed to which room. Then it's off to races as I start dragging talks into place in my grid area to make the first draft of the full schedule!

There are many dimensions to consider here, and like it or not, the choices that are made reflect the values of the event and its organizers. Factors I consider include:

  • the surveyed interest in the talk
  • the speaker's geographic origin
  • the subject of the talk
  • the speaker's first-time and underrepresentation status
  • the audience level of the talk

I start with the highest-voted talks first, placing them into the two larger rooms, then the lowest-voted talks into our smallest room. Some exceptions are made where we feel strongly about the value of a talk that might have been overlooked during the survey process; often we can get a bigger audience for and underappreciated talk by simply programming it into a bigger room, signaling that it we feel it deserves to be there.

To accommodate our speakers' travel needs, the speaker's geographic origin plays a big role in choosing which block a talk goes into. We basically have four big sections: Saturday morning, Saturday afternoon, Sunday early afternoon, and Sunday late afternoon. That late Sunday afternoon block is typically all speakers from Ohio or the surrounding area for whom driving home wouldn't be super inconvenient. Speakers from places that have reasonable Sunday evening flight options can go into the Sunday early-afternoon block. Speakers with fewer flight options usually end up on Saturday, with anyone from the west coast going into Saturday afternoon to minimize their time zone pain.

The topic of the talk is important too; attendees interested in a particular subject will probably appreciate being able to attend more than one talk in that subject area; it doesn't make sense to have all the machine learning talks at the same time since someone could only attend one. So I try to layer those into the schedule vertically, sequencing them so we can try to wring some synergy from them; a good example here was arranging two talks about Pytest such that the more introductory one came first and the more in-depth one came second, and both of them preceeded a tutorial on automated web UI testing. (The speakers even picked up on this after the conference, which was utterly gratifying!) Talks on topics which we feel are important or should be emphasized (such as--but not limited to--testing, security, ethics, humane cultural practices) often get programmed into the bigger rooms. Talks on more niche subjects, personal projects, and curiousities often end up in one of the smaller rooms.

From a diversity standpoint, it's not enough to have a particular quantity of underrepresented or first-time speakers, they have to be visible too; that means scheduling them throughout the conference instead of loading them up at the same time slot(s). I approach scheduling with the goal of having at least one underrepresented speaker in every time slot. I similarly thread the first-time speakers vertically through the grid so that we always have a balance of first-time and veteran speakers.

We've chosen not to prompt speakers to identify the audience level of a talk, so I take a bit of a guess at slotting a beginner-appropriate talk into every time slot. This way the less-experienced attendees will always have at least one good option to explore throughout the entire conference. I've gotten positive feedback on this, so I think we're doing okay on this front.

Once I've got all the talks placed into the grid in Padlet, I export a png and share it with the committee. (I do this png step to give us an approximation of version history.) We go through a few rounds of review and adjustment, and I second guess myself a lot, but it stabilizes pretty quickly. Once the three of us feel good about it, I run it by the conference chair for approval, and after that I use the CFP site to assign every talk to a room and time.

The last thing we do as a committee is a new activity we added this year: we create a public Google Calendar for the conference events. The entire conference schedule is added to the calendar, and speakers are invited to their own talks. This way it's easy for anyone who likes to manage their conference-going using calendaring apps to pick talks they want to attend, and speakers can effortlessly know where and when they need to be to deliver their awesome talks. We got a lot of positive feedback about this this year, so we'll definitely keep doing it as long as I'm involved.

After that we sit back and wait to hear from speakers who have questions or need help with something. It's a lot of work, but it's all worth it when the conference happens and completely exceeds our expectations as it has these past two years. It's really gratifying when it all comes together and you can feel the energy from the speakers and attendees. I'm extremely thankful to my fellow committee members, the organizers, our speakers, and our attendees for the roles they play in creating this very special event.

blog comments powered by Disqus

« Previous Post