Paul Groudas and Stuart Sierra are the two newest members of the Clubhouse team. Their addition marks Clubhouse’s deepening commitment to strengthening our infrastructure, enhancing site performance, and fully leveraging our investment in Clojure and Datomic.
We recently sat down with the two of them to learn more about their career trajectories, what they are working on at Clubhouse, and what they think is most important to focus on as Clubhouse continues to grow.
Paul: Technically, I’m the Clubhouse DevOps Engineer, but that’s a very overloaded and ambiguous term.
Stuart: And controversial.
Paul: Very controversial.
Paul: I’m doing a mixture of release engineering, infrastructure automation, and back-end engineering, but I think of it more as a support role for the other engineers. My priorities are one step removed from the product itself and aimed towards the engineering team of Clubhouse.
Prior to Clubhouse, I worked at Compass for four years. I helped found the company, where I was definitely a “wearer of many hats.” I focused mainly on building our core infrastructure and API, as well as building and maintaining our internal application platform as we grew.
At Compass, we had a microservice architecture that mapped to internal teams. We grew very rapidly, and by the time we had 50 engineers we had dozens of applications running on a dynamically-scaling application platform which took a little bit of work to deal with.
Here at Clubhouse, I’m hoping to keep us “lean and mean” as we grow. I’ve observed that Clubhouse is really entering that high-growth phase. It’s a good time to put some investment into operational automation and additional security. It’s also just fun to work with a small and energetic team!
Most of my consulting work was with development teams that had already adopted Clojure, Datomic, or both, helping them transition toward more scalable, robust architectures for their systems. I took prototypes and helped adapt them to operate reliably at larger scale.
Stuart: Yes, I almost forgot!
Stuart: I had just one brief session with Clubhouse as a consultant, answering questions about how to optimize a Datomic-based application. I’d seen the app but hadn’t had a chance to learn much about it yet.
One thing I did notice, right away, was that Clubhouse, as an application, is an ideal use-case for Datomic’s architecture. It’s a distributed application in which read operations are much more common than writes. It’s multi-tenant, which means it’s easy to divide up the load for optimal caching. And, of course, it needs to make use of history for things like activity feeds. After just a brief look, I could confidently say, “Yes, Datomic is absolutely the right choice for this application.
Paul: I came in with a bit of a different perspective. I’d known Kurt and Andrew, the founders, for a number of years. And I’d used Clubhouse during the early beta period, so I had some familiarity with the product.
But I came in with a critical eye at a different level. I was focused less on Datomic, because I was unfamiliar with it. I was much more interested in how Clubhouse was leveraging AWS. Amazon has really matured over the last few years, and they have such a broad portfolio of products that you can do the same thing a variety of different ways depending how your team operates, the size of your team, and the frequency of your releases.
The early phases of any engineering product generally involve a lot of changes of direction and technology choices. So the first thing I was trying to do was reverse-engineer not just “how we are doing things today,” but the “why,” so I could figure out what was important to keep, versus what just happened to be the landing spot at the time. I combined that with trying to project what our short- and medium-term infrastructure needs would be.
And that’s what I’m still focusing on. It’s a combination of engineering processes melding nicely with automation. We are outgrowing some Amazon tools that have served us really well. In particular, portions of our infrastructure are hosted on Elastic Beanstalk, which is a very opinionated Amazon platform. It’s very useful if you fit its mold but we are now growing in certain dimensions that no longer overlap with that, so we need to move onto some lower-level Amazon technologies. Doing that in a seamless way is our goal.
Paul: I think a lot of it has to do with ownership and responsibility. If you can bring good ideas, but then also support them and show your own personal buy-in about them, I think it’s easier to convince folks. It’s hard to win hearts and minds by just building something nice and putting it in front of someone. If you can build something and then use it and demonstrate how it helps, with clear communication, then you can get a lot more cultural buy-in.
It’s hard to win hearts and minds by just building something nice and putting it in front of someone. If you can build something and then use it and demonstrate how it helps, with clear communication, then you can get a lot more cultural buy-in.Paul Groudas, DevOps Engineer at Clubhouse
Stuart: I decided to leave consulting and join Clubhouse because I wanted the opportunity to work for a longer period with the same team and the same product. I wanted to participate in discussions about priorities, long-term goals, and strategic value.
Paul: Yeah, that’s one of the big advantages of joining a new team, or a new company like Clubhouse. You can grow with it.
Paul: The scope of problems that we are facing today is different than what we were facing six months ago. And in another six months, and another year, it will be different yet again. Growth naturally creates interesting challenges.
Clubhouse is at a really great point in that growth curve where there are going to be plenty of opportunities for good engineering solutions and creative ideas. The combination of technology and pragmatism pushes us to find solutions quickly as that hockey stick goes up.
Stuart: I feel the same way. I think I joined at the optimal time when things start to get interesting.
Paul: I’d say the lowest hanging fruit for me, personally, was codifying all of the state we had in AWS. We had grown organically, since the beginning of the company, in the number of AWS services that we were consuming. The services that we were using were still working well for us, so there was no immediate need to move quickly to different technology.
But there was a big need to capture the details of how those various services interact and how they are configured. Using the Amazon console for managing your infrastructure works if you just have one person handling a really small project. But as teams grow and the number of services grows, you inevitably spend longer periods away from any individual part of it. That means you can no longer rely on your memory to explain the “why” of how everything is configured. That becomes dangerous in terms of being able to makes changes with confidence and deal with incidents when portions of your system are having problems.
So, the first challenge I had was trying to pull together everything in Amazon and put that in the form of source code in Terraform. With that, we could have a source of truth that conveyed a bit more information to the team.
Stuart: I think the Clubhouse team has been using Datomic pretty well. I didn’t find anything where I immediately said, “Whoa, stop doing that!” But I was able to come up with a handful of optimizations that were easy changes to make right away. I also worked with Paul to get Datomic’s metrics integrated with the monitoring that Clubhouse already had set up.
Paul: That’s a challenging problem and a very good question. There are many dimensions to this, but one of the primary things is “How difficult or expensive is this problem to address today? And how will that change if we don’t do it right now, or in six months? And how do our growth metrics go into that?” The big driver is really, “How do costs change if you do them today or tomorrow?” Sometimes it gets more directly related to the product. There are improvements we may want to implement before launching certain features.
Stuart: Right. I was going to say the same thing about doing something now versus how much harder it’s going to be later. Also, how difficult something is going to be against how much potential benefit it has.
Paul: There are two that come to the top of my mind. Firstly, the unit economics and performance of our back-end servers. We want good financial economics, but we also want to maintain a high velocity developing the product itself. That gets more challenging when the infrastructure footprint is large. The approach we can take to fix that is to apply a few more layers of automation to minimize the amount of user attention it takes to get that software out there safely.
Then there’s the point of improving the unit performance of those servers, because if we can do that then the size of the infrastructure shrinks, or grows at a slower rate. That can extend our runway before we have to address this again.
Stuart: One big thing I learned while consulting was that you should try to take advantage of the things that are unique about the job you’re trying to do or the application you’re trying to build.
I think a lot of development teams miss out on this, maybe because they’re trying to copy the infrastructure of some other tech company they admire — often a much bigger one. Sometimes, teams will pursue an approach that is elegant in theory but does not map well to their application. Focus on the things that make your application unique and how that helps you. For example, Clubhouse is a multi-tenant application and we can take advantage of that to optimize how we approach load-balancing.
Sometimes, teams will pursue an approach that is elegant in theory but does not map well to their application. Focus on the things that make your application unique and how that helps you.Stuart Sierra, Clojure Developer at Clubhouse
Paul: Luckily, my entire career has overlapped with the transition from in-house infrastructure to “the cloud,” to the point where almost no one who knows what they’re talking about argues that the cloud is cheaper. The cloud’s not cheaper. With the cloud, you are trading infrastructure costs for developer flexibility. Because your engineers are the engine that drives a lot of product development, being able to have them focus more on things other than fixed infrastructure really helps.
The thing that’s really gotten better since 2007 is that managing infrastructure is more and more like writing good, well-tested software. In picking tools and technologies:
Stuart: And then measure again after you make the change.
As Paul, Stuart, and the rest of the team continue to refine the Clubhouse infrastructure to meet the demands of our ever-growing user base, we are learning a lot about the capabilities of our existing technologies and, when necessary, adopting new ones. We are excited to share what we’ve learned and will be blogging about our experiences and suggested best practices as we move forward.
How is your team preparing for growth? We’d love to hear from you on Twitter!