Level Design Survey

[Hourences]

Not entirely related to Source/HL level design, but interesting for you guys nonetheless, I hope
I am currently doing a big cross community survey on level design. How players, modders, and professionals experience it. Most focus is on FPS level design (aka like for the historical big three: UT/HL/Quake).
Nearly 600 people have participated so far, but I would love to hit the 1000 to get a really accurate picture of it.

# The survey is for both gamers and level designers/artists, and for amateurs and professionals.
# Please note that the term "level designer" is used broadly. "Level designer", in this survey, is defined as someone who designs and creates large parts of a virtual world. This can include the visuals or the gameplay, or both.
# The results are for personal use only. This survey is a personal initiative because I would really like to know how gamers, modders, and professionals perceive Level Design. For the community, by the community.
# The results will be announced mid september. I will post them here as soon as they are released!

Survey:
http://www.hourences.com/book/surveyactive.htm


Help me portray an accurate picture of the state of level design! The results will be read by a good number of professionals and modders across the communities.

Thanks!

 

[DjBourgeoisie]

done

 

[Pi Mu Rho]

and done.

 

[Ennui]

took this when you posted it on ModDB a few days ago. Good luck, still looking forward to the results!

 

[Majestic XII]

Done deal, love to see the results.

(nice to see you here btw, even if you're only stopping by )

 

[Crispy]

Sounds interesting, but I think your question format is a bit flimsy.

Instead of asking which of these annoying things in games annoy you, you could have asked respondants to place them in order of priority, or asked them to comment whether they find them 'very annoying', 'fairly annoying', 'annoying', 'not very annoying' or 'not at all annoying'.

This way you would get a much more accurate picture of what players and developers think about the different aspects of level design. In its current format the survey is in danger of confirming the obvious: that gamers like good graphics and strong and original gameplay in games, and find crap design frustrating.

Also, however much I may or may not agree with it, "Do you feel that the profession of a level designer is being threatened by the increasing trend of splitting the profession into many smaller jobs?" is a very leading question. You don't really define what you mean by 'threatened', and you don't define 'Level Designer' either. Results from this question will be inconclusive because of this.

"How do you feel about todays more complex tools?" - Obviously something more complex to the layman is going to make something harder. The question is worded badly.

---

(More tomorrow when I finish the survey)

 

[Hourences]

Thanks people

Crispy, the software did not support ordering the options, and I did not want to enable commenting because it is no fun at all to have to go through a 1000 comments, it should be automated.

I was also afraid of making the survey too long so I tried to cut down on questions. It is already 28 questions long right now. I realize that there are like 40 additional questions I could have asked, but hey, if this works out well maybe I will ask those other questions in a second survey next year or so.

And the splitting up question was pretty clear I thought. Do you think that the profession of a level designer is threatened because it is likely to be cut up into many different roles in the future because of the increasing complexity in game development? So do you think that this process will one day degrade the profession of a level designer so much that the term "level designer" practically disappears, or becomes a hollow shell?

It is very difficult to ask a complex question in a single sentence. That also goes for the complexity question. The tools did became more complex nowadays, but the support for those tools also increased, and the tools became much more userfriendly. Things are WYSIWYG nowadays. So the question is, did the increase in userfriendly tools and support made level designer easier for you, even though the tools did become more complex?
Or did the increase in complexity made it harder for you, even while there is a thousand times more support and better tools available nowadays?

I will clarify that in the survey.

 

[Crispy]

Oh gawd I just lost a massive post on why most of the questions are a bit flawed and the results can't be used very conclusively...

As for the profession of a level designer: In terms of the role, maybe. I can see that the role of someone who controls all aspects of level design (in the classic definition of the term) is probably going to disappear. In terms of how you worded it (that their "profession [...] is being threatened"), I don't think it is, since a classic level designer is competant in all fields and will not suddenly find themselves out of work. They will just have to adapt and specialise in a few fields. This is how I feel the question was worded a bit unclearly.

Honestly, I think the tools have made things harder, because you need to expend much more time getting shit done and learning the tools. Once you're experienced, fine, but getting there is a much harder process and even then you still take longer to make a level if you're doing it all yourself. We need more 'Lego'-style level design tools like you see for RTSs. You just don't have the same modularity and ease of use in FPS 3D level design packages.

Anyway I took the survey, but I definitely don't think that you could get an accurate picture of my opinion from the way the questions were worded and the options available. I'm happy to answer why at another time if you're interested, but I haven't got time to re-write the post right now.

---

What is the survey aimed to prove/analyse, by the way?

 

[Hourences]

Bump, the results are up!

http://www.hourences.com/book/surveyresults.htm

(Dozens of charts below all that text!)

Thanks to all who participated.

There were around a 1000 participants in this survey, of those, 58 percent were level designers or artists. The remaining 42 percent were gamers and developers involved in other aspects of development. Of the 58 percent, 27 percent were professional level designers (152 professionals).

38 Percent of the participants voted for Source as the preferred development platform by the way, which makes it the most popular platform. Unreal comes in second at 31 percent. All others are far behind (Quake only got 12 percent for example)

 

[Shinobi]

missed the survey

love your site though houruences

 

[Ennui]

Nice, cool to see you got so many responses, and an impressive write-up of the results. Cheers!

 

[Crispy]

Asked about what people regard as most important in a level, "Gameplay" unsurprisingly won with 812 votes. In second place comes "Visuals" (505 votes), closely followed by "Performance" (463 votes).

This highlights a loophole in the Survey format. If the total number of respondants was 1004, how can 812 say one element is the most important and more than double the remainder say that another element is the 'most' important. This is because multiple choice was allowed, which should never be possible in a 'What do you think is the ...' question.

This leads to some respondants interpreting the question as a 'choose one response only' and others selecting everything that they agree with. Results from this style of question are very unreliable in real terms.

The same issue can be seen in the 'most frustrating thing' in a multiplayer level, and in the single-player level equivalent.

Asked whether people felt that the increasing trend of splitting up the profession into many smaller jobs threatened the profession of a level designer, 37 percent believe that it does not. However, 31 percent of the people thought it did, and the remaining 32 percent voted for "Not sure".

I'd attribute this to the ambiguous nature of the original wording of the question.

And lastly, 68 percent believe that level design is one of the most complex parts of a game.

If the original question was worded like this, it's very likely that people who are experts at the technicalities of the discipline will say it is complex. It would probably have been better to do a multiple choice question regarding which areas were the most complex, but even that is skewed towards the experts in the field.

The survey should really have contained a question regarding where they heard about the survey. It is likely that posting a survey on predominantly Source communities and so on will get a good bit of support for Source. I'd be interested to hear which Unreal and so on communities were polled, and also the general split between who came from where, to see where the natural bias should lean to, and whether the results maintained that trend.

Overall the survey was a nice idea, but the questions should have been written better to eliminate any chance of misinterpretation and the options and option types should have been better suited to the questions to create more compelling arguments based on their results.

 

[Hourences]

It was posted on my site, and on a large Unreal forum (BU), and in the BU (largest Unreal community site) news. I am extremely at home in the Unreal community, so you can be sure that Unreal was well represented. Source does seem to be more popular, even more so when you consider that UT3 is relatively new, and still its modding community seem smaller than Source's. It should have easily scored more and it did not. Even if Unreal was underrepresented, it should have still scored a lot higher.

It is completely impossible to implement a "Who voted for what" thing. You would get 28 (questions) x 8 different communities) x 3 (pro/amateur/gamer) x 5 (for how much experience someone has)= 3360 answers/charts....
Waaay too many answers. No one would read that many.

A question "Where did you hear of this" would not help at all. All it would proof is that there are more Source people than Unreal people on this planet (for example), and that is exactly what the technology question indirectly also proves.

And I think it is rather obvious that a question is multiple choice if there is a checkbox rather than a radio box.

 

[Crispy]

It was posted on my site, and on a large Unreal forum (BU), and in the BU (largest Unreal community site) news. I am extremely at home in the Unreal community, so you can be sure that Unreal was well represented. Source does seem to be more popular, even more so when you consider that UT3 is relatively new, and still its modding community seem smaller than Source's. It should have easily scored more and it did not. Even if Unreal was underrepresented, it should have still scored a lot higher.

Well even if the trend seems to indicate this, the way the data was collected does not empirically prove this. It's just an observation, not a personal attack.

It is completely impossible to implement a "Who voted for what" thing. You would get 28 (questions) x 8 different communities) x 3 (pro/amateur/gamer) x 5 (for how much experience someone has)= 3360 answers/charts....
Waaay too many answers. No one would read that many.

I understand if this was a limitation of the polling system used, as opposed to the questions, but it wouldn't be impossible to at least ask an introductory "How did you hear about this survey? - BU/Half-Life2.net/etc./Other" question. Just giving a chart for what type of people answered the poll would at least give some information to go on as far as whether suppositions were met or not. If we knew to begin with (as an example) that Half-Life2.net had a bigger presence in the total amount of respondants, it would be less surprising if Source scored highly. Similarly, just having an idea of which websites had most representatives would give interesting data on (generally) which modding communities were more populated, which would arguably go some way to supporting a theory of which engine had the easier set of tools to work with and/or learn.

If you indirectly ask (i.e. do not record demographic numbers) a mixed group of Mexicans and North Americans whether they prefer home-cooked enchiladas or the Taco Bell equivalent and you get 70% saying they prefer Taco Bell, it could be because 70% of people who answered were North Americans and 30% were Mexicans, every person voting for what they are more familiar with. It could be that the split was 50-50 and Mexicans prefer fast-food to home-cooked. My point is that without the numbers you can never accurately claim that your survey proves one or the other.

A question "Where did you hear of this" would not help at all. All it would proof is that there are more Source people than Unreal people on this planet (for example), and that is exactly what the technology question indirectly also proves.

The tech question does not prove this. If there were two questions -one asking which level design tools you were familiar with (multiple choice), and another asking which you preferred (one choice only)- that would form a more reliable set of data. The first question would show which tech had more user penetration via its games or free tools, and the second would actually show which was preferred overall. If the first question showed a vast majority of people having experience with Source and nothing else, it would only go to show that if Source was preferred it was because it got there first. 'Getting there first' is the most reliable way of convincing someone you have the superior product (see "Positioning: The Battle for the Mind" if you want to read a book on this topic). If you get there first, most often unless you give them a good reason your 'customers' will not feel any urgency to even try out another product, they will simply be happy with what they have because it works for their needs and they are comfortable with it, whether its a brand of cola or 3D level editing software. This is why software retailers do trial editions and cola manufacturers give away freebies at public events.

And I think it is rather obvious that a question is multiple choice if there is a checkbox rather than a radio box.

It is obvious, but it is also obvious in my eyes that a "What do you prefer" question should only allow one response, not multiple. If I ask you if you prefer Coke, Pepsi or Dr. Pepper, most people will choose one option. If you give them a multiple choice with the same question, some will answer with their preference and others will choose multiple 'preferences' which distorts the data produced.