Open-Sourcing our judging process

Words by Christopher Heymann — 1kx

A fair, innovative and correct judging process was a top priority when we started planning the hackathon. Here is the story of how we came up with it, and a breakdown to help you understand what you will face, and what you will go home with after ETHBerlin:

The basics:

• This criteria only applies to the Open Track Bounty. Sponsored prizes have their own criteria
• 36 judges divided into 12 groups of 3.
• Each submitted project will have 3–5 minutes to present their idea and 5–3 minutes for Q&A and feedback from the judges.
• Each valid team submission will be reviewed by two groups, to minimize subjectivity and to maximize feedback.
• Judging groups will be conformed by:
• at least one technical person able to evaluate the technical merit and the codebase of the submission
• at least one person with VC experience to give feedback on the practicality of the project
• The wildcard judge, who can be a project founder or an academic, to give input from an additional angle
• Judging will be based in the following criteria:
• Technicality: How hard was it to build this and how does the quality of the codebase compare?
• Originality: Is this something completely new or are there already projects with grant funding working on a similar scope??
• Practicality: How would this submission survive in the real world, outside of the crypto community? Or, if the project were to be specifically targeted to the crypto community, how correctly targeted is it?
• WOW Factor: What else is there about the submission? Has this project and/or team defied expectations, based on timing, efforts, etc?i?
• We deliberately chose not to judge project aesthetics, as we want the teams to focus on tech and ideas — however without deprecating UX, a project able to deliver groundbreaking UX, will be evaluated accordingly yet separately from this process (see github for the Community Improvement Prizes) Pretty slides and graphics skillz should not be your concern — at least for the weekend.

The following multiples will be applied per judge role to each criteria:

• Technical Judge: Technicality x 1, Originality x 0.75, Practicality x 0.25, Wow x 0.75
• Business/VC Judge: Tech x 0.25, Originality x 0.75, Practicability x 1, Wow x 0.75
• Wildcard Judge: all criteria x 0.6875

We, the ETHBerlin judging committee, took inspiration from a lot of our predecessors while defining how submissions will be judged at our hackathon. Here is a little story on how it developed and where we ended up:

(When I write we, I mean specifically:
Mareen Glaske from Gnosis, Maria Paula from Golem, Lasse Clausen from 1kx, Benjamin Bollen from OST and Christopher Heymann from 1kx.)

One key thing ETHBerlin is about, is learning and growing –that is also at the heart of the judging process we have designed. The hackathon should help you identify your strengths and weaknesses and receive constructive and honest feedback from your peers, mentors and judges.. Let’s face it: here can only be a few people on the final stage, however,no one should go home empty-handed, a hackathon is a learning experience — without diplomas, but a solid one.

Past hackathons and learnings

Our key metric is to give super helpful feedback, and for this, we needed to put some extra effort. We enquired both judges and hackers from previous ETHGlobal hackathons and came to two main conclusions:

• Submissions were usually judged in under 5 minutes.
• That time was spent all presenting ideas and answering questions, and almost no feedback round took place.

To improve this we added 3 minutes to these 5 minutes of presentation and Q&A, during which the judges can focus on really helping out the brave projects that submitted their work. .

Judging method

Another angle that we felt could be improved was subjectivity. In most previous hackathons every submission was judged by only one group. This might create scenarios in which multiple high quality submissions are penalised by the luck of draw because they were all assigned to the same group of judges.

To tackle this,we will make sure that each submission will be judged by two different groups of judges, which should reduce subjectivity to a practical minimum. This has the welcomed benefit, that it will double the feedback that every submission receives from the judges!

In total we are expecting that every submission will have a grand total of 16 minutes of judging time that way.

Judging criteria

In previous hackathons five criteria were taken on consideration:, technicality, originality, practicability, aesthetics and Wow! factor.

We will give this twist:the aesthetics criteria has been discarded to allow attendees concentrate their efforts on the real substance of their submission.

Here is a breakdown of what we are looking for in each respective criteria:

Technicality

• How technically-challenging is this submission? Considering the number of developers who have contributed to it.
• How is the quality of the produced code? Are there obvious code-smells and likely security holes?

Originality

• Usability aside, is the idea proposed something that hasn’t been explored enough and rightfully deserves more attention?
• Are we witnessing the birth of a completely new use-case for tokens or smart contracts?

Practicability

• Apart from the idea, does this submission have the potential to be impactful for other people from the crypto community?
• Does this submission have the potential to disrupt existing economies?

Wow! Factor

• How has this project defied expectations?I.e. did they do an extraordinary job producing code with just a few coders or is the scope of the project more ambitious than for a hackathon yet it was still delivered?

The points for each criterion range from 1 to 3 points, so in practice, if none of the above questions could be answered, the submission should receive 1 point for each criterion. If one of the above questions could be answered with yes, this should in practice result in +1 point for the respective category.

TL;DR? WELL, WE DECIDED TO VALUE ORIGINALITY AND WOW’NESS OVER TECHNICALITY. YOUR PROJECT NEEDS TO TRANSCEND. WE WANT YOU TO BLOW OUR MINDS.

Details for submissions and the basic submission requirements will be announced soon. You can submit for both open track and sponsored prizes — always.