There's been some Twitter back-and-forth about the way we've been running the State of CSS survey, and since this raised valid concerns I thought I'd address them here.

The criticism in a nutshell: our survey is biased against the women of the CSS community.

Exhibit A: Survey Demographics

The first argument in that direction is that our respondent demographics are pretty biased: in 2019, 85% of our respondents were male.

Now I'm not sure what data source we would use to determine the "real" demographic breakdown of the CSS ecosystem, as the Stack Overflow survey for example has a 91% male population – but then again it doesn't focus specifically on CSS.

This is why I tend to push back when people make the jump from seeing a skewed result set to automatically assuming there are problems with the survey's methodology. This doesn't mean the survey doesn't have issues, just that a skewed chart by itself without more context is not evidence.

So what are those issues? I believe they lie in the way we distribute the survey.

Our main channels are social networks (Hacker News, Reddit, Twitter, etc.), which carry their own preexisting bias, and our email list of respondents from past surveys – which propagates any existing bias from one year to the next.

The consequence of all this is that our distribution methods are indeed biased towards a male audience.

Exhibit B: Survey Resources

The other thing that was pointed out is that our Resources section includes no women, which is also true.

But if you take a closer look, you'll see that it's pretty clear we've tried to stay away from individual blogs or sites altogether, precisely because once you start including Persons A and B, people will rightfully ask why Persons C and D were not part of the list.

The two exceptions are David Walsh, because his blog often features guest writers and is not quite a typical personal blog, and Wes Bos (in the "Sites & Courses" section) just because of the sheer size of his audience.

But the bottom line is that the Resources section can definitely be improved to better reflect the CSS community, and I think we will probably start by reconsidering our decision to exclude personal blogs for next year (leave your suggestions here).

Exhibit C: We’re JavaScript Developers

The first survey we launched was the State of JavaScript survey back in 2016, which we then followed up with the first State of CSS survey in 2019. So during those first three years, we did accumulate a very JavaScript-focused audience.

So it's understandable why some people might think we are outsiders barging in without trying to represent the real CSS community.

Moreover, as JavaScript has come to take more and more space on the front-end over the past couple years, it’s understandable that the CSS community might be afraid of this “invasion”. Especially if this threatens the things that make the CSS community so great, such as the fact that it’s traditionally been a more welcoming place for women and minoritized demographics.

Then again, the reality is that CSS is changing, and that there is no impartial authority to decide who represents “the community” and who doesn’t. A CSS working group member working on the next specification and an open-source developer hacking on a new CSS-in-JS library are both advancing the CSS state of the art, in their own ways.

Yet this “pro-JavaScript” bias in our audience is real, and one way we hope to mitigate it in the future is to give you tools to filter survey results by respondent source — such as people who found it through Twitter, CSS-Tricks, or our pre-existing mailing list. In the meantime, we do make all of our data public if you want to dig through it yourself.

So… We're Biased?

At this point you might be surprised if you expected this post to be an impassioned defense against the critics. Turns out I agree with them!

What I don't agree with on the other hand is the assumption that these biases stem from laziness, negligence, or lack of caring. The truth is just that this is a hard problem to solve!

We ask many different people to give us their feedback each year (and we also directly asked the 10,000 subscribers to our mailing list, too) and we’re hugely grateful for their help and suggestions; but as you'd expect many more are also too busy with their own projects to spend time and effort helping us.

And while suggesting we "involve more women" can seem like a no-brainer, this also puts even more burden on the very people who are being let down in the first place… We never want "involving women" to become a way to offload our own responsibilities onto them!

So we do our best with the time and resources we have. And if that best is not good enough for everybody, all I can say is that it’s not for lack of trying.

Making Things Better

All that being said, if you do want to help, what can you do?

First, you can give us feedback in anticipation for next year’s CSS survey. And while pinging us on Twitter is a good start, leaving us a comment in the dedicated GitHub threads for the State of CSS and the State of JavaScript surveys is even better since it'll ensure we can refer to it one year from now when we're working on the next edition of a survey.

Barring that, you can also give us time. Building the infrastructure to run these surveys takes time, and so does building relationships with key members of the community. Please remember that this is only the second time we’re running the State of CSS survey, and it’s been a tough year for everyone, us included.

If this post can show you one thing, I hope it’s that I’m not interested in making excuses or sweeping our issues under the rug. I’m interested in improving the survey and making things better, and hopefully so are you!

Note: I'm taking a break from social media so I won't respond to tweets or DMs, but I'll try to keep an eye on the comments here.

Thanks to Stephanie Walter and Christian Oliff for reviewing drafts of this post.