Blog: How we improved feature flag resiliency

[neilkakkar]

Changes

Please describe.

Add screenshots or screen recordings for visual / UI-focused changes.

Checklist

• Titles are in sentence case
• Feature names are in sentence case too
• Words are spelled using American English
• I have checked out our style guide
• If I moved a page, I added a redirect in vercel.json

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
posthog	✅ Ready (Inspect)	Visit Preview	Sep 8, 2023 9:33am

[neilkakkar]

Okay, rough structure is there.

I could use some feedback now @andyvan-ph @joethreepwood @liyiy @EDsCODE on content, structure, and if this all makes sense.

And @ellie @hazzadous on accuracy of the small infra things I've mentioned here.

[ivanagas]

A bunch of edits.

I think it is important to tighten up the intro and get to the "meat" of the article faster (as it is really interesting and I want to make sure as many people as possible get to that point)

[liyiy]

I like the light touch that a small joke about asteroids brings to the piece. I think it'd be even better if it was only contained in the last 1-2 sentences or only referenced 1-2x at most instead of 3-4x? 🤗

[neilkakkar]

LOL agree I think I liked it so much I forgot I already included it 😂

Have removed and combined the above sentence into one.

[liyiy]

Overall awesome work! Really helps to paint a picture of all the work the team has done in the past couple of months to improve feature flags 🎉

[liyiy]

I'm not sure what images we would put here, maybe some screenshots of latency improvements from our grafana as an example? Just to split up the text a bit and make it more (visual) reader-friendly

[neilkakkar]

I think we could lift this with one or two diagrams / flow charts that visualize what you're writing about. Any thoughts on what these could be? (attached an example from a recent newsletter illustrating what I mean)

Hmmm, good question, thinking about this, but nothing great comes to mind. Maybe a flow chart of how things are setup and where the borkages happen?

Like, a sample request flow?

Request comes in ---> Django server for feature flags ---> fetch feature flag definitions from redis

  --- first option---> try evaluating without database --> return result if evaluated
  ---second option--> Pgbouncer (connection pooler) -----> Database to fetch person properties

And the arrows going to redis and pgbouncer are sources of problems & latency.

[ivanagas]

How's this? Could potentially do ones for Partial flag evaluation and database down if good

[neilkakkar]

Ooh yes this is great!

[neilkakkar]

I'd just tilt them to make the flow clearer.

PostHog server on the left, and top

SDK in the middle, both height & width wise,

and client at the right, and bottom.

Should read more naturally, with the arrows going something like:

[ivanagas]

Partial flag evaluation

[ivanagas]

[ivanagas]

Another option

[neilkakkar]

yes 2nd option is very nice

[andyvan-ph]

@neilkakkar + @ivanagas: Thanks both for the graphic stuff. Have added both and done another light polish pass on the copy.

I was doing a mental Hacker News pre-mortem and I think the only thing this is missing is... evidence. We say we've made flags faster / more reliable, but there's nothing to back that up.

@neilkakkar is there something we can near to the end here? I don't think it need to be super in-depth, it just needs something to prove it out.

[neilkakkar]

hmm, we don't have our latency logs anymore, because this was > 3 weeks old. So we can't show before & after, but I guess we can show current latency times

and then the status page: https://status.posthog.com/uptime/1t4b8gf5psbc?page=3 and how incident rate has gone down.

Will add a blurb to the end when I'm back tomorrow, thanks!

[neilkakkar]

just added this section as an appendix