Video Transcription

one of the big success factors here atSpotify is our agile engineering cultureculture tends to be invisible we don'tnotice it because it's there all thetimekind of like the air we breathe but ifeveryone understands the culture we'remore likely to be able to keep it andeven strengthen it as we grow so that'sthe purpose of this video when our firstmusic player was launched in 2008 wewere pretty much a scrum company scrumis a well-established agile developmentapproach and it gave us a niceteam-based culture however a few yearslater we had grown into a bunch of teamsand found that some of the standardscrum practices were actually getting inthe way so we decided to make all thisoptional rules are good start but thenbreak them when neededwe decided that agile matters more thanscrum and agile principles matter morethan any specific practices so werenamed the scrum master role to agilecoach because we wanted servant leadersmore than process masters we alsostarted using the term squad instead ofscrum team and our key driving forcebecame autonomy so what is an autonomousAutonomous Squadsquad a squad is a smallcross-functional self-organizing teamusually less than eight people they sittogether and they have end-to-endresponsibility for the stuff they builddesign commit deploy maintenanceoperations the whole thing each squadhas a long-term mission such as makeSpotify the best place to discover musicor internal stuff like infrastructurefor a be testing autonomy basicallymeans that the squad decides what tobuild how to build it and how to worktogether while doing it there are ofcourse some boundaries to this such asthe squad mission the overall productstrategy for whatever area they areworking on and short term goals that arerenegotiated every quarter our office isOur Officeoptimized for collaboration here's atypical squad area the squad memberswork closely together here withadjustable desks and easy access to eachother screens they gather over here inthe lounge for things like planningsessions and retrospectives and backthere is a huddle room for smallermeetings or just get some quiet timealmost all walls are whiteboards so whyis autonomy so important well becauseit's motivating and motivated peoplebuild better stuff also autonomy makesus fastby letting decisions happen locally inthe squad instead of thea bunch of managers and committees andstuff it helps us minimize handoffs inwaiting so we can scale without gettingbogged down with dependencies andcoordination although each squad has itsown mission they need to be aligned withproduct strategy company priorities andother squads basically be a good citizenin the Spotify ecosystem Spotify isoverall mission is more important thanany individual squad so the keyprinciple is really be autonomous butdon't sub optimize it's kind of like ajazz band although each musician isautonomous and plays his own instrumentand listen to each other and focus onthe whole song together that's how greatmusic is created so our goal is looselycoupled but tightly aligned squads we'renot all there yet but we experiment alot with different ways of gettingcloser in fact that applies to mostthings in this video this culturedescription is really a mix of what weare today and what we are trying tobecome in the future alignment andAlignment and Autonomyautonomy may seem like different ends ofa scale as in more autonomy equals lessalignment however we think of it morelike two different dimensions down hereis low alignment and low autonomy amicro management culture no high levelpurpose just shut up and follow ordersup here is high alignment but still lowautonomy so leaders are good atcommunicating what problem needs to besolved but they're also telling peoplehow to solve it high alignment and highautonomy means leaders focus on whatproblem dissolves but let the teamsfigure out how to solve it what aboutdown here them low alignment and highautonomy means teams do whatever theywant and basically all run in differentdirections leaders are helpless and ourproduct becomes a Frankenstein we'retrying hard to be up here alignedautonomy and we keep experimenting withdifferent ways of doing that soalignment enables autonomy the strongeralignment we have the more autonomy wecan afford to grant that means theleaders job is to communicate whatproblem needs to be solved and why andthe squad'scollaborate with each other to find thebest solution one consequence ofCrosspollinationautonomy is that we have very littlestandardization when people ask thingslike which code editor do you use or howdo you plan the answer is mostly dependson which squad some do scrum sprintsothers do Kanbansome estimate stories and measurevelocity others don't it's really up toeach squad instead of formal standardswe have a strong culture ofcross-pollination when enough squads usea specific practice or tool such as getthat becomes the path of leastresistance and other squads tend to pickthe same tool squads start supportingthat tool and helping each other and itbecomes like a de facto standard thisinformal approach gives us a healthybalance between consistency andflexibility our architecture is based onover a hundred separate systems codedand deployed independently there'splenty of interaction but each systemfocuses on one specific need such asplaylist management search or monitoringwe try to keep them small and decoupledwith clear interfaces and protocolstechnically each system is owned by onesquad in fact most quads owns severalbut we have an internal open sourcemodel and our culture is more aboutsharing than owning supposed squad onehere needs something done in system Band squad two knows that code bestthey'll typically ask squad two to do ithowever if squad two doesn't have timeor they have other priorities then squadone doesn't necessarily need to wait wehate waiting instead they are welcome togo ahead and edit the code themselvesand then ask squad two to review thechanges so anyone can edit any code butwe have a culture of peer code reviewthis improves quality and moreimportantly spreads knowledge over timewe've evolved design guidelines codestandards and other things to reduceengineering frictions but only whenbadly needed so on a scale fromauthoritative to liberal we'redefinitely more on the liberal side nownone of this would work if it wasn't forthe people we have a really strongMutual Respectculture of mutual respectI keep hearing comments like mycolleagues are awesome people often givecredit to each other for great work andseldom take credit for themselvesconsidering how much talent we have herethere is surprisingly little Eagle onebig aha for new hires is that autonomyis kind of scary at first you and yoursquad mates are expected to find yourown solution no one will tell you whatto do but it turns out if you ask forhelp you get lots of it and fast there'sgenuine respect for the fact that we'reall in this boat together and need tohelp each other succeed we focus a loton motivation here is theexample an actual email from the head ofpeople operations hi everyone ouremployee satisfaction survey says 91%enjoy working here and 4% don't now thatmay seem like a pretty high satisfactionrate especially considering our growthpane from 2006 to 2013 we've doubledevery year and now have over 1200 peoplebut then he continues this is of coursenot satisfactory and we want to fix itif you're one of those unhappy 4% pleasecontact us we're here for your sake andnothing else so good enough isn't goodenough half a year later things hadimproved and satisfaction rate was up to94% with this strong focus on motivationit's no coincidence that we have such anawesome reputation as workplacenevertheless we do have plenty ofproblems to deal with so we need to keepimproving okay so we have over 50 squadsspread across four cities some kind ofstructure is needed currently squads aregrouped into tribes a tribe is alightweight matrix each person is amember of a squad as well as a chapterthe squad is the primary dimensionfocusing on product delivery and qualitywhile the chapter is a competency areasuch as quality assistance agilecoaching or web development as squadmember my chapter lead is my formal linemanager a servant leader focusing oncoaching and mentoring me as engineer soI can switch squadswithout getting a new manager it's apretty picture huh except that it's notreally true in reality the lines aren'tnice and straight and things keepchanging here's a real-life example fromone moment in time for one tribe and ofcourse it's all different by now andthat's ok the most valuablecommunication happens in informal andunpredictable ways to support this wealso have guilds a guild is alightweight community of interest wherepeople across the whole company gatherand share knowledge within a specificarea for example leadership webdevelopment or continuous deliveryanyone can join or leave a guild at anytimeguilds typically have a mailing listbiannual on conferences and otherinformal communication methods mostorganizational chart are an illusion soour main focuses community rather thanhierarchical structures we found that astrong enough community can get awaywith an informal volatile structureif you always need to know exactly whois making decisions you're in the wrongplaceone thing that matters a lot forReleaseautonomy is how easily can we get ourstuff into productionif releasing is hard we'll be tempted torelease seldom to avoid the pain thatmeans each release is bigger andtherefore even harder it's a viciouscyclebut if releasing is easy we can releaseawesome that means each release issmaller and therefore easier to stay inthis loop and avoid that one weencourage small frequent releases andinvest heavily in test automation andcontinuous delivery infrastructurerelease should be routine not dramasometimes we make big investments tomake releasing easier for example theoriginal Spotify desktop client was asingle monolithic application in theearly days with just a handful ofdevelopers that was fine but as we grewthis became a huge problemdozens of squads had to synchronize witheach other for each release and it couldtake months to get a stable versioninstead of creating lots of process andrules and stuff to manage this wechanged the architecture to enabledecoupled releases using chromiumembedded framework the client is nowbasically a web browser in disguise eachsection is like a frame on the websiteand squads can release their own stuffdirectly as part of this architecturalchange we started seeing each clientplatform as a client app and evolvedthree different flavors of squads clientapp squads feature squads andinfrastructure squads a feature squadfocuses on one feature area such assearch this squad will bill ship andmaintain search related features on allplatforms a client app squad focuses onmaking release easy on one specificclient platform such as desktop iOS orAndroid infrastructure squads focus onmaking other squads more effective theyprovide tools and routines for thingslike continuous delivery a be testingmonitoring and operations regardless ofthe current structure we always strivefor a self-service model kind of like abuffet the restaurant staff don't serveyou directly they enable you to serveyourself so we avoid handoffs like theplague for example an operation squad orclient app squad does notput code into production for peopleinstead their job is to make it easy forfeature squads to put their own codeinto productiondespite the self-service model wesometimes need a bit of think betweensquads were doing releases we managethis using release trains and featuretoggles each client app has a releasetrain that departs on a regular scheduletypically every week or every threeweeks depending on which client justlike in the physical worldif trains depart frequently and reliablyyou don't need much upfront planningjust show up and take the next trainsuppose these three squads are buildingstuff and when the next release trainarrives features a B and C are donewhile D is still in progress the releasetrain will include all four features butthe unfinished one is hidden using afeature toggle it may sound weird torelease unfinished features and hidethem but it's nice because it exposesintegration problems early and minimizesthe need for code branches unmerged codehides problems and is a form oftechnical debt feature toggles let usdynamically show and hide stuff in testsas well as production in addition tohiding unfinished work we use this toa/b tests and gradually roll outfinished features all in all our releaseprocess is better than it used to be butwe still see plenty of improvement areasso we'll keep experimenting this mayseem like a scary model letting eachsquad put their own stuff intoproduction without any form ofcentralized control and we do screw upsometimes but we've learned that Trustis more important than control why wouldwe hire someone who we don't trust agileat scale requires trust at scale andthat means no politics it also means nofear fear doesn't just kill trust itkills innovation because if failure getspunished people won't dare try newthings so let's talk about failureactually no let's take a break get onyour feet get some coffee let this stuffsink in for a bit and then come backwhen you're ready for party