What would make you turn in your eval?

We’re working on lots of details for Southwest Fox right now. One of them is figuring out what to offer attendees to ensure that evaluation forms get turned out.

As you can see from some of our previous posts (like The Economics of Conference Food or They like us! They really, really like us!), we really pay attention to the evaluations that our attendees submit. We crunch the numbers they give us and we read all the comments more than once. We may not be able to implement all the changes that people request, but we do at least consider them.

We provide every attendee with a packet of evaluation forms at registration. The packet includes an overall conference evaluation and a short evaluation for each time slot. Unfortunately, lots of people don’t turn in their evaluation packets.

Last year, we were able to increase the yield of evals significantly. Microsoft had given three MSDN licenses to each developer MVP to give away. A number of our speakers and several other friends of Southwest Fox donated some of those gift licenses for raffles. We reserved the last three for a drawing including only those who submitted evaluations.

This year, we don’t have a big-ticket item like that to give away. So our question is what would make you turn in your eval? We know that some people will turn in evaluations with no prize at all. If you’re not one of those, help us figure what to use an incentive.

We want to keep making Southwest Fox better and to do that we need feedback. So give us some ideas on keeping those cards and letters coming.

Conference Hotel Room Feedback

In general we get a lot of positive comments on the hotel rooms for the conference, but this year there were a few people who suggested we look into a new location because there were problems with their rooms. Your feedback is real important to us. Last year (based on positive attendee feedback and our experience) we locked into the Arizona Golf Resort and Conference Center for 2008 and 2009. After next year’s conference we have an option to look at other locations if it makes sense to do so.

I would like to address some of the trends in the comments:

1) Free wireless not available in the rooms, only the conference center. True, and it is one of the reasons we pay close to $400 a day ($1600 for the four days) for connectivity to the Internet in the conference center. I agree that spending $10 a day per person for the “facility wireless” is a bit of a hassle, but fortunately it is available if you needed it. We tried to negotiate this last year, but the wireless is provided by an outside provider and it was non-negotiable. Good news for next year: free wireless should be available throughout the conference center including hotel rooms. The conference center started testing it at this year’s conference. Unfortunately it was not quite ready when I tried to use it.

2) Golf course is not open. True and very intentional. While we are sure there are golfers who attend the conference, you actually get a discount while the golf course is closed. I don’t have the exact numbers, but if I recall correctly we are saving more than $20 a night by booking our conference when the course is closed. Last year they surprised us by opening the course early, and this year they opened the course the day after we finished. If you are an avid golfer, the front desk is more than happy to book you some time on other courses in the area and I am sure they would shuttle you if necessary.

3) Hotel Rooms are far from conference meeting rooms. Obviously true for some people. Last year my room was out in the middle of the golf course and I liked the walk. I can see some people don’t and that is understandable. What you might not know is the facility has people standing by 24 hours a day to take you to and from your room by golf cart. We probably need to mention this in our materials and communicate to the hotel desk to make this known when you check-in.

4) Sounds like some of the hotel rooms had issues with appliances, shower heads, and a couple with insects and one with a mouse. The feedback also indicated the hotel staff resolved the issues, which does not surprise me. Others reported the rooms were outdated. Unfortunately we don’t know what rooms had problems because you did not mention it in the evals. If you would not mind letting us know which rooms (send an email: INFO at SWFOX dot NET, or give me a call at 586.254.2530) we certainly will provide these details to the hotel and express our concerns for next year.

We are sending the conference center your feedback as part of our feedback to them. I can tell you that they listen to us (and you too), and have done their best to respond to our needs and to our constructive criticism. We certainly hope you also took an opportunity to do the same.

Where was KOKOPELLI?

Happy New Year everyone.

One of our questions on the evaluation form asked people to rate KOKOPELLI. One of the themes we got in the responses is “What is KOKOPELLI?” and “Where was KOKOPELLI?”

Apparently we did not do a good job making the Southwest Fox schedule helper applet from Dave Aring public enough on the Web site, in our email announcements, and the RSS feed. You might not be aware that we had it loaded on the kiosk computer we had set up at the registration table. The majority of people who used it gave it a high rating, which is good. What is not good is many people did not get to take advantage of the tool.

Looking on the conference Web site I see it mentioned once on the News page. This is where the only link is. I am hopeful we will get a second chance from Dave to better promote KOKOPELLI. If he does, I promise we will make it more apparent how to get it, and where you can find it.

Southwest Fox 2008 is this year. Only 287 days until we meet in Mesa!

The Space Race

One complaint we saw a lot on the evaluations was that a room was overcrowded for one session or another. This is actually a hard problem for us to solve.

Conference room layouts fall into two basic categories: those (like this year’s Southwest) where all the rooms are the same size, and those where session rooms have different sizes. Each type presents issues for scheduling, but on the whole, it’s a lot easier when all the rooms are the same size.

When rooms of significantly different sizes are used, whoever makes the schedule not only has to try to figure out which sessions fit together to give everyone something to attend at each slot, but also which sessions will draw big crowds and which will be attractive only to a select group.

Sometimes, that’s easy. I was pretty sure that Christof’s “Dark Side of VFP” session would draw a huge audience. He’s a great speaker, it’s a sexy title, and people know that he knows more about what’s going on inside VFP than just about anyone outside Microsoft. (Now there’s a thought—wouldn’t it be cool to have had Calvin Hsia, VFP’s Lead Developer, in the room for that session!)

On the other hand, I was surprised by the SRO crowd for Cathy’s “Outfox the VFP Report Writer” session. I knew she’d give a good session, and there’d be plenty of meat, but I didn’t expect it to have such wide appeal. Shows what I know.

If we’d had rooms of different sizes, I would have put Christof’s session in the biggest room, but not Cathy’s. In fact, the first time those two sessions were given, they were side-by-side.

With same-sized rooms, we don’t have to try to figure that out. On the other hand, we don’t have the flexibility of multiple sizes. It’s a bit of a catch.

The one area where we do have flexibility on this is in how we have the rooms set up. I think pretty much everyone prefers classroom style, with tables and chairs, over theater style, with only chairs. But of course, we can fit a lot more people in in theater style. (According to the conference facility, the rooms we used can fit about twice as many in theater style.) We chose to split the difference, putting a few tables in the front part of the room, and then several rows of chairs in the back. Each room was initially set up for 50 people. (That changed as people moved chairs from room to room to meet needs. One thing you can always count on about the Fox gang is that they’re not afraid to move furniture.) Four rooms of 50 seats each seemed pretty good for 150 attendees plus 16 speakers (with four speaking at any given time).

For next year, we’ll look hard at the set up of the individual rooms and perhaps use fewer tables and more rows of chairs. I think we’ll also try to keep sessions we expect to be very popular out of room 4, the one at the far end of the hall, because traffic in and out of there was difficult.

They like us! They really, really like us!

The title says the most important thing we learned from the Southwest Fox evaluations. Most attendees were very happy with the conference. Of 92 people who filled out the overall conference portion of the evaluation booklet, 88 said they were likely to come to Southwest Fox again in the future. That’s an incredibly high number, but matches what people told us in person.

Although the overall tone of the comments was positive, nothing else had quite that high a score. Nonetheless on a five-point scale (from “Disagree” or “Poor” or “Not important” to “Agree” or “Excellent” or “Very important,” depending on the question), the average score was above 4 on every question except one.

What was that one? “How important is the facility and location in your decision to come to Southwest Fox in the future?” The average score for this one was 3.18, just a hair above neutral. Some people went farther in their comments, indicating that the room is irrelevant to them as long as it has a bed.

As is usually the case, we learned more about what people thought from the comments than from the numerical answers. A few issues were mentioned on lots of evaluations.

First, we blew it on coffee. In trying to keep costs under control, we decided to provide a full (rather than continental) breakfast, buffet lunch and an afternoon snack. At the last minute, we added coffee and tea to the first morning break. (Since none of us are coffee drinkers, we hadn’t fully thought this issue through until we were on site.) Good thing. Even with coffee and tea at breakfast and the morning break, a lot of people commented on the need for coffee all day. We’re definitely revisiting this question for next year.

We also blew it on the pre-con day with no drinks (other than water) or snacks. I think I can state that we’ll certainly have coffee and tea available; we’ll look at the question of snacks, as well.

Another frequent comment was a request for power strips in the session rooms. An increasing number of people expect to run their notebooks in sessions (especially since we provide free wireless in the conference center). Because our session rooms were all part of a larger ballroom, most of the available outlets were in the back of the room. People wanted more. This is an easy-to-solve problem.

We had a lot of comments about the temperature in the session rooms. But it wasn’t as simple as “it’s too cold” or “it’s too hot.” It varied by room, time of day, and number of people in the room. We actually went to the facility staff a number of times during the conference to have them adjust the temperature, but there were enough variables that it was hard to get it right.

My favorite comment, which appeared on a fair number of evaluations was along the lines of “this session paid for the conference.” That’s a goal we always have as speakers, and anytime we see that comment, we’re thrilled.

More to come.

Reporting on the overall conference data

Reporting on the overall conference evaluations was both easier and harder than reporting the speaker evaluations. Easier because we needed only a single report, with no need to break things down by speaker. Harder because instead of three open-ended questions, there were 12.

My first inclination was to handle the comments as I had on the speaker evaluations, with a separate cursor and detail band for each. But 12 detail bands seemed excessive. More importantly, I was feeling too lazy to write the code for the 12 queries and set up the 12 relations and so forth.

Instead, I decided to use the solution I’ve been using for years for simulating multiple detail bands. Create a single cursor with all the data in the right order, and an extra field that identifies the “detail band” to which a given record belongs. That still called for an enormous UNIONed query and I still felt too lazy to write it. So I decided to let the query write itself.

I wrote code to create a cursor with one record for each open-ended question. Then, the code loops through that cursor and uses textmerge to create the query that collects data for that question.

With that out of the way, the rest was easy. I wrote a query to compute the averages for the various numerical questions. For some questions, we actually wanted to see the breakdown of answers, not just the average, so the query handles those with SUM(IIF(, 1, 0)).

Here’s the complete program:


SELECT CNT(*) as nEvalCount, ;
SUM(IIF(lWeekend, 1, 0)) as nWeekend, ;
SUM(IIF(lWeekend, 0, 1)) as nWeek, ;
AVG(nKeyLength) as nKeyLength, ;
AVG(nKeyConten) as nKeyConten, ;
AVG(nKeyNext) as nKeyNext, ;
AVG(nConfRooms) as nConfRooms, ;
AVG(nWireless) as nWireless, ;
AVG(nHotel) as nHotel, ;
AVG(nTopics) as nTopics, ;
AVG(nLevel) as nLevel, ;
AVG(nSpeakers) as nSpeakers, ;
SUM(IIF(lKokopelli, 1, 0)) as nUsedKoko, ;
SUM(IIF(lKokopelli, 0, 1)) as nNoKoko, ;
AVG(nKokoUse) as nKokoUse, ;
SUM(IIF(lNextYear, 1, 0)) as nNextYear, ;
SUM(IIF(lNextYear, 0, 1)) as nNoNext, ;
AVG(nMaxHotel) as nMaxHotel, ;
SUM(IIF(nMaxHotel=1, 1, 0)) as nHotel1, ;
SUM(IIF(nMaxHotel=2, 1, 0)) as nHotel2, ;
SUM(IIF(nMaxHotel=3, 1, 0)) as nHotel3, ;
SUM(IIF(nMaxHotel=4, 1, 0)) as nHotel4, ;
AVG(nFAcility) as nFacility ;
FROM ConfEval ;
INTO CURSOR ConfSummary

CREATE CURSOR OpenEnded (cQuestion C(100), cField C(10))
INSERT INTO OpenEnded ;
VALUES ("What did you like about the conference?", ;
"mLikes")
INSERT INTO OpenEnded ;
VALUES ("How can we improve the web site?", ;
"mWeb")
INSERT INTO OpenEnded ;
VALUES ("How can we improve the registration process?", ;
"mRegister")
INSERT INTO OpenEnded ;
VALUES ("How can we improve the conference check-in?", ;
"mCheckin")
INSERT INTO OpenEnded ;
VALUES ("How can we improve the variety of topics presented?", ;
"mVariety")
INSERT INTO OpenEnded ;
VALUES ("How can we improve the evening sessions/activities?", ;
"mEvening")
INSERT INTO OpenEnded ;
VALUES ("How can we improve other things?", ;
"mOther")
INSERT INTO OpenEnded ;
VALUES ("What else should the keynote include?", ;
"mKeyOther")
INSERT INTO OpenEnded ;
VALUES ("Speakers recommended?", ;
"mNewSpeak")
INSERT INTO OpenEnded ;
VALUES ("Topics recommended?", ;
"mNewTopics")
INSERT INTO OpenEnded ;
VALUES ("Why not attending next year?", ;
"mWhyNot")
INSERT INTO OpenEnded ;
VALUES ("Other comments?", ;
"mComments")

#DEFINE CRLF CHR(13) + CHR(10)

cQuery = ""
nItem = 1
SCAN
IF NOT EMPTY(m.cQuery)
cQuery = cQuery + CRLF + "UNION ALL ;" + CRLF
ENDIF

cThisQuestion = ALLTRIM(OpenEnded.cQuestion)
cThisField = ALLTRIM(OpenEnded.cField)
TEXT TO cThisQuery TEXTMERGE NOSHOW
SELECT RECNO() AS nRec, ;
<> AS mComment, ;
"<
>" AS cWhich, ;
"<>" AS cQuestion, ;
<> as nSort ;
FROM ConfEval ;
WHERE NOT EMPTY(<
>) ;
ENDTEXT

cQuery = m.cQuery + m.cThisQuery
nItem = m.nItem + 1
ENDSCAN

cQuery = m.cQuery + CRLF + "ORDER BY nSort, nRec INTO CURSOR curComments"

EXECSCRIPT(cQuery)

SELECT curComments

REPORT FORM ConfEval PREVIEW

The report looks simpler than the speaker evaluation report. All the average and break-out data is in the title band, which is set to appear on a separate page. Then, there’s a single group, based on the nSort field. The group header contains the cQuestion field and the detail band contains the mComment field and a dotted line, so we can see where one person’s comments end and the next begins. Here’s a picture of the report:

Next time, some comments on the actual evaluation data.

Processing the Evaluations

I didn’t expect helping to run Southwest Fox to give me a chance to learn new things about VFP. But speaker coordination was one of my responsibilities, so I took the lead in handling the evaluation data. On the plane coming home, I designed a couple of tables to hold the data and started working on data entry forms.

We gave each attendee an evaluation booklet with an identical section for each time slot, plus an overall conference evaluation at the end. (One of the things I learned doing the data entry was that we should put the overall conference evaluation at the front of the booklet, not the back.) At the end of the conference, we divided the evaluation booklets into three equal piles; we each took a pile home with us.

Because the data entry would be distributed and we wanted things to be simple, I chose to go with a nearly flat file design. I created two tables, one to hold evaluations of the individual sessions, with one record for each session evaluated, and another to hold the overall conference evaluation data, with one record per evaluation booklet. We already had tables for the speakers, topics and timeslots, so I put foreign keys to those into the session evaluation table.

I created a data entry form for each table, keeping things as simple as I could, but with an eye toward streamlining data entry. One thing that made creating these forms easier was that we only needed to add records, not to edit them.

The real fun came in creating reports once we’d done the data entry. We needed two reports, one for the session evaluations and one for the overall conference evaluations. Each posed interesting problems. The key issue in both cases was that we had lots of free-form comments in the evaluations and needed a way to combine that data with the averages for the numeric questions.

For the session evaluations, we also needed to give each speaker his or her own evaluations and no one else’s. I started, as I pretty much always do for reports, by working on queries to pull out the data to report. I needed to average each of the numeric ratings for each repeat of each session, as well as count the number of evaluations submitted. No problem; a simple grouped query handles that:


SELECT iTimeSlot, iSpeaker, iTopic, CNT(*), ;
AVG(nPrepared) as nPrepared, ;
AVG(nKnowledge) as nKnowledge, ;
AVG(nInteresti) as nInterest, ;
AVG(nMatchDesc) as nMatchDesc, ;
AVG(nValuable) as nValuable, ;
AVG(nRelevant) as nRelevant, ;
AVG(nAgain) as nAgain;
FROM SessionEval ;
GROUP BY 1, 2, 3

To add the speaker’s name, the session title and the time the session was given, I used that query as a derived table (one of the features added in VFP 9 that’s really grown on me over time) inside another query that joined the look-up tables with these results:


SELECT SessAvg.*, Speaker.cFirst, Speaker.cLast, ;
Topic.mTitle, TimeSlot.tStart ;
FROM ;
(SELECT iTimeSlot, iSpeaker, iTopic, CNT(*), ;
AVG(nPrepared) as nPrepared, ;
AVG(nKnowledge) as nKnowledge, ;
AVG(nInteresti) as nInterest, ;
AVG(nMatchDesc) as nMatchDesc, ;
AVG(nValuable) as nValuable, ;
AVG(nRelevant) as nRelevant, ;
AVG(nAgain) as nAgain;
FROM SessionEval ;
GROUP BY 1, 2, 3 ) SessAvg ;
JOIN Speaker ;
ON SessAvg.iSpeaker = Speaker.iID ;
JOIN Topic ;
ON SessAvg.iTopic = Topic.iID ;
JOIN TimeSlot ;
ON SessAvg.iTimeSlot = TimeSlot.iID ;
ORDER BY cLast, cFirst, iTopic, tStart ;
INTO CURSOR SessionAvgs

With that data in hand, I looked at the harder part. For each session, we’d provided three places to write comments:

  • What did you like about this session?
  • What didn’t you like about this session?
  • Other comments

I wanted to group all the answers for each question for a given session together, so the speaker would see all the positives, then all the negatives, then any other comments.

Making this work gave me my first chance to use VFP 9’s multiple detail band reports in a production situation. I wrote three separate queries to pull out the answers into separate cursors, and related each of those cursors to the summary cursor, SessionAvgs. Here’s the query and relation for the things attendees liked about a session:


SELECT iTimeSlot, iSpeaker, mLikes, RECNO() as nRec ;
FROM SessionEval ;
WHERE NOT EMPTY(mLikes) ;
ORDER BY 1, 2 ;
INTO CURSOR curLikes
INDEX on PADL(iTimeSlot, 2) + PADL(iSpeaker, 2) TAG TimeSpeak

SELECT SessionAvgs
SET RELATION TO PADL(iTimeSlot, 2) + PADL(iSpeaker, 2) INTO curLikes

Then, I started to design the report. I added two groups: iSpeaker, the speaker’s primary key; and transform(iSpeaker) + transform(itimeslot), the combination of speaker and timeslot. I set the inner group to start on a new page. (In writing this, I realize that the outer group was unnecessary.)

I put the session-level information in the group header of the inner group, laid out pretty much as it looked in the evaluation booklet.

Next, I added two additional detail bands to the report. I set each of the three detail bands to be driven by one of the cursors containing the comments, and I gave each detail band its own header and footer bands (a very cool feature). I put the question for each band into the detail band header. The detail band itself contained a single field, set to float and stretch, with an expression like this:


transform(curLikes.nRec) + ": " + curLikes.mLikes

Why include the record number? To let speakers match up likes, dislikes and other comments from a particular attendee.

Once I got all this working, I thought I was done, and sent the data, the code and the report off to Rick and Doug for comments. Doug took my queries and modified them to generate rankings for both speakers and sessions. We decided that speakers might want to know their own rankings, so I incorporated Doug’s code into the program, and added a line to the group header to show that information.

I also added some code to generate PDFs, one per speaker. I tried both FRX2Any and XFRX, and found that for this particular report, XFRX handled things better. (I’m using each of the two for different client projects at the moment.)

Again, I thought I was done, but when I started writing a cover email to go to the speakers, I realized that, along with the rankings, it would be useful for the speakers to see the overall averages for each of the numeric items, so I added another query to calculate those, and added that data to the group header, as well.

Here’s the final report as it looks in the Report Designer:

Since this is already way too long for a blog entry, I’ll write about the report for the overall conference evaluations separately. Over the next week or two, I’ll also talk a little about what people had to say about Southwest Fox and the sessions.